Collaborate - Ignore certain asset folders

Hello, I have used Collaborate for a while now. Mostly, it has been quite a great experience. As time passed by, due to numerous and frequent changes, our project started to gain in size... quite a lot. As the project grew in size, we noticed quite a slowdown when using Collaborate (especially when loading scenes) and we actually hit the offered free storage. I am currently cleaning up the project, deleting unused assets, scripts and so on. What i was wondering if it's possible to exclude/hide certain asset folders entirely from Collaborate, so we can sync only the relevant folders (which are well bellow the size of the entire project)? . I tried using hardlinks, but haven't had any luck, collaborate would still pull all the changed files from the entire assets directory.
We are using Unity 5.4

 

Hi,
We worked on a Collabignore solution that will be available in our 5.5 experimental (preview) build. Unfortunately, it won't be ported for 5.4 versions of Unity.

Out of curiosity, can you give me a little more information? I would be interested to know the size of your project along with the amount of files. Also are you using Windows or OSX? I want to recreate that kind situation.

thanks!

 

Hello again, sorry for the late reply. Our project had ~= 15 gb of assets (mostly large textures, scenes), after cleanup (removing unused scenes and graphics - animations/textures) our project now has approx ~= 8 gb and ~= 5000 files, it seems it did regain some of it's usual speed and we can use collaborate again, yay . I am using windows 10

 

Glad to hear that.
I'll still do some investigations on project size issues. Recently we had a fix for files moves being slow but I'm not it was or will be ported to 5.4.

 

Our team recently ran into a need for a collab ignore file as well, We're running 5.5, is this feature implemented yet? Thanks!

 

@cubedparadox Actually, yes! As of yesterday our Preview build with these new features was released. Check out the forum post here for link to download and information on what we added.

https://forum.unity3d.com/threads/c...d-available-now-exciting-new-features.443913/

 

Hi, I'm also encountering an issue with this, but instead of any of the Asset files, it's actually the project Temp folder instead. Basically what I did was I had a Execute-In-Edit-Mode script that downloads assets from a server to apply to game objects during edit mode(so that I can visually my scene objects instead of having to always run the game to view them). I assume those files are stored in the Temp folder and are somehow included into the project when submitted to Collab.

EDIT: Looks like I was mistaken, those assets are all stored in the scene file itself.

 

Hi, I'm not sure exactly how this would help because these assets are downloaded from a script on edit-mode to the client device's temporary folder instead of being stored in the project folder, but because the game objects in the scene itself has a reference to those objects outside of run time, it somehow stores them into the scene file itself as well(thus increasing the size dramatically to the range of a few hundred megabytes)? Unless I misunderstood I'm not sure how to exclude specific assets from being saved into the scene file itself when publishing.

 

Basically the files that are downloaded from the server are image/texture/sound files(in this case, png and wav files), which are then appropriately converted and assigned on runtime/edit-mode to game objects(canvas sprite images, in this case). The thing is that because this occurs during edit-mode as well, the references to the temporary assets are never removed and are somehow stored in the scene file itself. The size of the scene file basically jumped from a few hundred kilobytes to a few hundred megabytes. So essentially with this I'm forced to ALWAYS upload a few hundred MB sized scene file to Collab instead of a few hundred KB one.

Another thing to note is that it may also be possible that other things could be contributing to the file size increase as well, as I am using singleton managers with DontDestroyOnLoad enabled in the scene(one of which handles the asset download and allocation), though highly doubt they are the cause, but it is worth mentioning just in case.

 

Ah well I kinda reverted my project to before implementing the edit-mode scripts to prevent this issue. As other tasks currently take priority at the moment I will not be reimplementing the scripts until after the end of the month as we're on a tight schedule for our game's alpha release. I will definitely post back here though once I do.

 

I would like to renew the original request for a "Exclude from Publishing" as it's also part of an issues me and my team facing right now.

As thing are currently, we're currently forced to use an external folder (we usually call "WIP") that store all the project work-in-progress files. Those are large files such as PSD, SPP (Substance Painter),SBS's References files and many other things we do share periodically, but in partiality and mostly through Google Drive especially because we can manage what is send or not.

The thing is that we would share those files through Unity Team (Collaborate + Cloud Build) if we were able to exclude files from publishing because, otherwise, it's inefficient.

Let's put a scenario to explain why the current inefficiency.

Situation :
The team has developed a project and is around 50% done. It's a mobile game and there currently about 5GB of files in total. 3 people works on those 10GB of files which includes all raw files (work-in-progress files). The actual project's size is around 300MB if you strip all the raw files and, instead, use finalized and optimized images/audio/mesh files.

Each of the 3 team members works on different part of the project. Each day, they publish 2 to 3 times their changes. A problem of time needed arise when whenever they publish changes done in the raw files as some of those files are heavy. For example, it's not rare that a SPP file and all its references is around 150MB for each files. Whenever there are changes done in that file and, because of another change that everyone else has to receive, that's 150MB of upload and download time required per file for everyone changed. (In terms of average, a Substance Painter file is approx. 150MB per material in the file. If it has 10 materials as it's filled with many models, that's approx. 1.5GB for a single SPP file)

That's especially true with the recent addition of updated compatibility with using raw .SPP directly in Unity for development and preview testing.

Now, whoever works in 3D and in graphics in general is well aware that saving (quick CTRL+S) is something you do as often as necessary... after almost every important and/or long changes. It's almost up to be called a mental sickness in a way. That's because most of us might have lost either hours of work or even just 20 minutes through the event of a crashing or corrupted software. The 2 most used shortcuts keys used by graphic-related artists are CTRL+S and CTRL+Z and most artist who still uses their keyboard even with a pen pad (Wacom or whatever) keeps their pinky on the CTRL key all day long.

That's something real and because we save files changes so often... if we're doing changed on some heavy graphical files that are still unfinished and, then, someone in the team request that we fix something small on an existing model or UI... We have to publish both the small fix requested and the unfinished heavy stuff taking both member's time (and others too in the end as they might also have to publish/download the fix).

By allowing files to be manually excluded from the publishing, it allows each team members to oversee their own version controls over their files. If a work-in-progress file isn't finished and is still edited, it shouldn't have to be published.

Same with smaller things like scripts. It's a lot quicker to upload and download so the time isn't much of an issue with those kind of files. But still, if I were to, for example, work on a scene with a bunch of newly made scripts for the content of that scene and it's overall unfinished yet, I could want to not publish that scene and its content while, yet, be able to apply any fixes requested by the rest of the team on the existing and already published content.

This is something I have personally faced with a project I'm currently working on. To put it simple, I couldn't publish my changes for 2 months because I couldn't exclude some heavy or incomplete files from the publishing process. I had to almost completely finish a whole concept into a almost-finished piece of the game just to push that Publish button while mostly only giving out update through our communication software (Discord) with some screenshots now and then. I had to do it like this because the changes I made and raw files amount in the 1.5GB-2GB. Imagine running a version control out of that amount of changes and data. Each time I would have published, that would have been 60MB-250MB of changes through only a small amount of files.The finished stuff only amount to about 80MB and that's what I'm publishing to my team.

 

@Max_Bol The 2 features we have so far that attempt to address this issue are: Partial Publish and the .collabignore file. It seems like these would address your scenario, but maybe I'm missing something?

 

@dtakahas Thanks a lot. Strangely, I wasn't able to find those pages at all through common google search or the search bar in the documentation. (And I did search a lot). Google doesn't return those pages at all unless you search for their exact name, but instead give out links toward pages that might contain obsolete information (which some date from near the beginning of 2017).