Excluding folders from build

Hey all

I just posted a suggestion for something I would find super useful and I'm sure many others would:
https://feedback.unity3d.com/suggestions/allow-tagging-folders-for-exclusion-in-builds

An example of where it would be useful is, I have a project that I want to release on the Amazon App Store and Google Play. If I add plugins to enable various store / platform specific features then the only control I have is to say these plugins are Android. Thing is, I don't want any trace of the Amazon plugins in the Google Play build and vice versa. Now imagine that I also want to release a paid version and a free ad supported version of the game. I want the paid version to have no on-line requirements, and no funky permissions due to ad libraries or social services. How to do that:

1. Maintain separate scenes for each and trust that the build process will chop out anything that isn't referenced. I'm totally happy relying on this for assets like textures, audio, etc. Not so much for plugins. I've recently released a game that used a popular Android plugin for a few minor platform specific features. In the Google Play developer dashboard I tick that this game does not use adverts, and it immediately pops up a warning saying "well your APK contains ad libraries, so we ain't inclined to believe ya".
2. Maintain multiple projects and use version control to sync up shared folders. This is how I handled my last one. I use externals to reference shared folders containing all my game's scripts and assets that aren't plugin based. It works, but it's super clunky. It also means it takes up way more space on disk and requires way more time transferring changes to & from VC. It falls apart at the slightest provocation (possibly because I use SVN for version control), and it's not hard to get my shared folders out of sync either through forgetfulness or some other user error.
3. Write build scripts that copy/paste or rename folders so they are excluded. No. Just no.

If there's some editor feature that I've missed that makes this super easy then please chime in. In that case I'll delete the suggestion and throw a party for whoever sets me straight.

Anyhow, how I'd see it working:

• You define tags somewhere in the project / build settings (probably Project Settings > Editor). Similar to how you define custom layers or object tags.
• The inspector for a folder in the project view has a drop-down where you select which tags apply to each folder (defaults to all). Saves to meta as you might expect.
• In the build settings you have a drop-down where you select which tags to build.
• Additional member in BuildOptions for people that want to control the build pipeline through scripts.

From the outside looking in, it seems like it would be a relatively simple feature to implement (I may be very wrong). It would make a huge difference to the neatness, flexibility and maintainability of my projects, though.

Chris

 

AFAIK, there is no built-in or a plugin that does this at the moment.

We have also faced a similar requirement working on our previous project: our game had to be built in a few different variations. By variation, i mean different platforms, and different configurations per platform.

This meant we had to configure assets, scenes, plugins, etc per every build variation.

There are different solutions your can do to work around this issue, which will probably cover most of what you need. For example - for scripts, it's pretty easy to differentiate code per variation using scripting defines (e.g: #if AMAZON).

What you'd like to do (include/exclude certain folders) is not possible with how Unity works. I've read about some solutions that suggest that if you move content into a particular folder (.tmp i think) it will be excluded but i never tried this.

What we implemented was a system we called "Workspaces". It used symbolic links, which is supported both on Windows and OSX. Essentially, we created a bunch of folders outside of the Assets folder. Each folder was assigned a "tag" like you suggested. When setting your project to a certain "workspace", the system we built would create symbolic links inside Assets to all folders that belong to that tag, so the editor "thinks" these folders are part of the project and include them. At any point you can switch to a different workspace and it would reset the symbolic links and include another set of folders.

It worked quite well though it did had a few shortcomings.

Maybe this gives you an idea on how you can solve this issue yourself for now, until Unity will support this out of the box.

 

Hence why I posted an idea. If it seems interesting and useful to you then give it a vote so that the Unity team can see it would make a difference.

I'd be interested to hear what the shortcomings are? I tried similar with SVN externals, but it was far too unwieldy to really use in production. Switching between configs was a complicated error prone process that could take quite a bit of time. Plus when you actually move files / folders around switching back to the editor then means it has to reimport all scan through all those files and reimport a number of them.

Thanks for the suggestions anyhow. I'll have a dig into symbolic links and .tmp folders and see what difference they make.

 

See here about using the WebPlayerTemplates folder for scripts you don't want to compile. Probably not what you want but worth knowing.

 

The shortcomings (as far as i can remember them) were partly related to our implementation (bugs, etc).
Also, we had to decide whether we commit the currently active "workspace" to source control, or whether we should leave that as a setting that is relevant per developer.
The solution itself worked - it did not involve any copying of folders; it only created a link to the actual folders. No copying/moving was involved.

After modifying your current workspace, Unity would reimport all things that were changed, since as far as the editor is concerned, the contents of the projects got updated. This can be a lengthy process, all depending on the content you include per platform.

For us, the game itself did not change apart from a view popups / images. You would generally want to keep most of your project common between all different flavours, and only change very small parts of it, in order to keep maintenance to a reasonable cost (of developer's efforts and time).

 

I just upvoted the suggestion. I have a pretty good workaround but it's not ideal.

Currently, I handle this through organizing my hierarchy so that conditional compilation is accomplished by disabling entire hierarchies from the top level. You can add/remove a tilde (~) from the end of the directory names to suppress visibility to Unity. Unity will entirely ignore all variety of assets below those directories, scripts, geometry or anything else.

This isn't hard to make part of your build process if you have a little bit of automation. When you add the tilde, all scripts in the project are immediately recompiled. To remove it you'll need to go into your platform's file manager, though, unless you build a tool for it.

I use it to simplify building for the Moverio glasses (on Android) and the HoloLens (on UWP).

There really should be a tool inside Unity to do this though.

 

Thanks for the tips, EddieOffermann. It helped me get over my hurdle of deploying to multiple platforms. I agree that Unity should be doing this by default and invisible to us (aside from elements we add to our scenes).

For those who want to know more about the tilde (~) method, you should review this document:
https://docs.unity3d.com/Manual/SpecialFolders.html

 

Thanks - that tilde trick is great to know!

 

How does the tilde trick work with revision control? Wouldn't each rename count as a change?

 

I wanted to rename as part of a build process, so it wouldn't affect source control for me. I would rename, do the build, then rename back.

 

We tested hiding folders again in 5.4 and 5.5 a while ago using the method of renaming with ~ but Unity's metadata didn't work properly and case-related errors appeared in the Editor (OSX) console about half the time making it unusable. We filed a bug which was accepted by QA but hasn't gone any further yet.

 

I'm interested in this too.I would love for Unity to have an option on folders like it has on plugins, allowing to exclude/include them depending on the platform !

 

I was shocked, but since 5.3.2 you can exclude files and folders from build with "~" suffix.
See 5.3.2 Release Notes (687655), and the updated Unity - Manual: Special folder names.

So one can create an Editor Tool that does some renaming magic related to some tagging mechanism.

 

I actually tried using that but it didn't work out so well. I advise you to do the same.

For example - our project took a very long amount of time to switch platforms due to many art assets. I've tried moving some of the unused stuff to a folder that starts with ~ to avoid having those assets imported, but Unity still processed them (at least, from what i can remember).

We ended up not using this approach at all.

 

I'm using it for plugin development just fine: native iOS / Android plugin projects can live within folders starting with ~.

For asset management there should be different answers. Starting with "why you keep unused stuff around" at all?

 

Consider a case where the project contains multiple "variatons" or "flavors". For example : paid vs. free versions, different platforms (under Android there could also be Google play / Amazon versions), etc.

Having a solution that lets you configure different "active" assets according to the active variation is a nice feature.

 

You'll need an Editor Tool for this anyway.
Moving files around or suffix folders with ~ makes not much difference.

 

of course, but the underlying mechanism for this tool isn't the ~ prefix for folders. We eventually went with something else which i previously wrote about in this thread.

 

Actually I did not read the entire thread, but the title.

 

They have to END with "~" not start
As the quoted page states:

"Files and folders which end with ‘~’."

Besides that, does anyone knows if those files get still included in the build (like the streaming assets) or if they get ignored for builds as well (which is what I'm looking for)?

 

Checked in Unity 5.6.3f3
file Assets/Resources/Textures/LoadingScreen/LoadingScreen~.png
is included in build ((

 

It's out dated, but I want to leave my case. According to Unity document, I could exclude my 'Resources' folder when building, by moving 'Resources' folder into 'Editor' folder. It solved my headache problem.

 

You probably should have - it's only 22 postings, not one of those 22 pages threads ;-)

@EddieOffermann posted about the tilde(~)-solution here, and @Beguiled followed up with a link to the documentation.

But I'll admit - when I first saw this thread I also missed that important piece of information and went right to the entry on Unity Feedback to support it ... but maybe this was a good thing because I eventually found a solution which works extremely well for me and is not only compatible with, but actually based on version control (so if you're not using version control, or, more specifically, if you're not using Git for version control, you're probably out of luck for this one):

Git has kind of an esoteric feature called sparse-checkout. What this does is that it lets you define only specific folders to check out from a repository. Depending on how you have set up your project, you'd then either use this (tell it which folders you want), or use /* and then !/... for the folders you want to exclude.

In my situation, I have one main development machine with the complete project, and one iMac build machine, and another PC for doing Android builds. But obviously, you could achieve the same on just a single machine by checking out the project folder three times, and then synchronizing the folders via git. I have been using this workflow for about two weeks now and am super-happy with it.

Just be a little careful when you already have the project checked out, and it's a very large project (I'm not that proficient in using git-command line and caused a situation where I had to re-get the whole project which takes almost an hour, and I was kind of scared I messed up that project on that machine during that time ;-) ).

I started out simply deleting the folders and ignoring the huge number of deleted files in version control. sparse-checkout then remedied that.

So here's how I went about it:

1. Obviously, you need to use Git for version control, with git-lfs for the large files.
2. In a "safe clone" (that you feel comfortable messing up), delete the folders that you believe you don't need.
3. Put the path of each folder into the sparse-checkout file, which is located in <projectFolder>\.git\info. It has no file-extension, the full name is really just sparse-checkout and it's a simple text-file. See below for an example.
4. In a command line or terminal, go into your project folder and execute the command git config core.sparsecheckout true
5. Then, do git reset, then do git checkout.
6. Problem solved
7. Copy your sparse-checkout into the root of the project - that way, you can version control it, and copy it to the "active location" in clones of the project where you need it. Just don't forget doing git config core.sparsecheckout true in those clones - ideally before checking everything out.
8. You might even have different versions of sparse-checkout for different project configurations.
9. You might need to cut some dependencies by using conditional compilation (I use this in a pretty large project with lots of dependencies). The only thing to be careful with here is making sure you don't accidentally revert such changes while experimenting (I had 20,000 files deleted at one point - very easy to miss real changes when you revert, then)

Sparse checkout example:

Code (csharp):

1. # see also: https://stackoverflow.com/questions/51241736/how-can-i-hide-not-ignore-deleted-files-in-one-clone/51243634#51243634
2. # put into \.git\info
3.  
4. # Only on build machines for iOS and Android (no need for CSCore and Morph3D there)
5.  
6. /*
7. !/Assets/Morph3D/
8. !/Assets/Morph3D.meta
9. !/Assets/MORPH3D/
10. !/Assets/MORPH3D.meta
11. !/Assets/Plugins/3rdParty/CSCore/
12. !/Assets/Plugins/3rdParty/CSCore.meta
13.  

What does this do? /* includes everything, then the !/Assets/... lines include what I need. As mentioned above, you could also just include the folders you need, whichever is more convenient in your case.

 

Even though this thread is around 2 years old, it's still relevant today. I actually had to revisit this recently in a project and i believe this can actually be solved in a nice way using editor scripting.

I implemented a small system for supporting what @Survivalist-Games initially wanted. In case someone is still interested i can share some details. I think it can also make up for a nice asset store package for people to buy (or get for free, we'll see).

 

Sounds cool, I'm definitely interested. One nice thing about physically excluding the folders (see my posting above) is that the "sparse projects" are much smaller and therefore much easier to work with, and I see anything that might break immediately during edit time (some things still only become obvious when doing an actual build but only comparatively few).

But if you can handle @Survivalist-Games idea using editor scripting and make it open source, I'd be happy to add code that auto-generates the sparse-checkout file from that information (could also go the other way round and apply a tag by parsing a sparse-checkout file).

 

I am actually using "sparse checkout" but for totally different reasons. My main problems with this approach are that
1. It is specific to a particular VCS system (git), although it may exist in other systems as well.
2. You really have to re-structure your project to efficiently support this. this may not always be possible (for example - plugins that get installed under a specific folder, for example).

Ideally, i would just want to "not checkout" the "Amazon" folder and be done with it. if i have to start picking folders that will not be checked out, it gets slightly more complicated.

 

I'm interested. I did find a solution too but I don't think it is a really a good one.

I have created an "_Assets" folder in which I have added a folder for each platform. Then I have created symbolic linked folders, for each of one of those, inside the "Asset" folder.

To exclude folders, I did a little script using the CompilationPipeline event, and renamed the symbolic linked folders with the "~" character regarding of the value of EditorUserBuildSettings.activeBuildTarget.

@liortal Can you share some details of your solution ?

 

@liortal I'd love more details too, or would be interested in a paid asset. Would be very useful.

 

This is a very old thread. The system i described back then worked for us (in 2016), what more info would you like to get?

It would be best if you described your exact requirements or what you're trying to achieve, then maybe i can help you out better in finding the exact solution.

 

@liortal I guess we're just following up on your post from this July; "I implemented a small system for supporting what @Survivalist-Games initially wanted. In case someone is still interested i can share some details. I think it can also make up for a nice asset store package for people to buy (or get for free, we'll see)."

I tried the tilde hack but it doesn't really work for me.Broke stuff as some files/assets were excluded while others were not. I found it easier to just cut and paste entire folders out of the Asset folder. Not ideal, so when you mentioned you had a solution that might even be turned into an asset, I got interested

The reason I'm looking for something like this is that as I'm optimizing I'm finding a bunch of stuff from stray packages are making it into my build. Nested Resources folders, shaders, that kind of thing. I'd love to be able to disable things without having to uninstall, to try different variations and compare performance.

 

i think a native solution to exclude some Resources folders at build time via script is very important.
i subscribe to this thread just for updates

 

I think that some "workaround" can be developed that is fairly simple (until official Unity support rolls out, if ever).

@StefanoCecere let me know if you're interested in hearing more about that (are you only after removing a set of pre-selected folders from the build?)

 

+1 come one Unity, that should be very easy to implement and very useful for us!

 

+1 we need

 

+1 we also needed

 

+500 we need

 

yes please

 

+100

 

Hi, according to the documentation, adding a hidden flag for asset folder also prevents it from being included, just like adding ~ to the end of the name. Using hidden flags may be the more convenient option as the directory won't be moved back and forth in version control.

That said, excluding assets from builds and then reverting them back can cause problems. As an example, I excluded UMA 2 Avatars from my project's server build and used conditional compilation (#if ... #endif) to remove my UMA-related classes from the server build. Later when I continued working with the client build again, I noticed that component references were broken from prefabs to the UMA-related components that I had not compiled for server build. I had to rebuild the entire Library folder to get them back (Unity 2019.1.8f1) -- took about 30 minutes.

To keep this simple, I have found that it works best to have "the full master version" in version control. From that you can make platform specific project directories that exclude some parts. As an example, you could make a server build directory that excludes Audio, Animations and Textures with rsync as easily as this:

mkdir ../[game]-server ; rsync -r -v --delete Assets ProjectSettings Packages ../[game]-server --exclude 'Assets/Audio' --exclude 'Assets/Animations' --exclude 'Assets/Textures'

 

OK, I had a solution in Unity 2017 LTS but currently I'm moving our project into Unity 2018 and this solution doesn't work properly anymore. It's related to excluding folder, cloud build, and multi-platforms.

What I did is write a few scripts about changing settings, changing icons, change compile symbols, and renaming folders that needs exclude. Those folders are simply renamed to "Folder~" in order to be excluded. And it works in cloud build. At the same time, compile symbols and potencial compiling errors MUST be handled correctly, in compile symbol define, and in scripts too. It worked on Cloud Build. And I need to make sure those folders are renamed back immediately after builds in order to prevent errors. All folders are in "Assets" folder.

But I'm moving the project to Unity 2018 LTS, it seems the behaviour is changed, and I have trouble setting compile symbols during cloud build process. Even the compile symbol is set, the script still consider itself that the symbol is NOT SET. It works on local but not on cloud build.

Well, still need this feature.

 

One more from me ! Please !!!!

 

+1

 

+999

 

+1

 

+1

 

This would be a great feature!

In my case, I built support into my game to create and edit levels from the game itself. I have thumbnails I load up from the resources directory that I use during this in-game development process. I use a preprocessor define to know whether I need to include the various options for doing this.

It'd be great if I could have a pre-build function that checks if that directive is defined, and if not, it marks my particular folder to be not included. For now, I don't see any way around deleting it before I make a real build (there's a lot of content in there not worth ignoring).

Thanks, hope this gets into the engine soon!
-Matt

 

+1

We need something like this:

https://docs.unity3d.com/ScriptReference/Build.IFilterBuildAssemblies.OnFilterAssemblies.html

 

The 2.0-preview package does allow users to put files outside of the "Assets" folder in course control, see:
https://forum.unity.com/threads/cho...es-1-2-16-and-2-0-preview-15-packages.683197/

There's a more general workflow where a unity project may be considered as a subset of the entire workspace under version control that we're looking to solve.

 

I don't use collaborate and the solution to this problem it shouldn't be tied to that service.

 

+100 sucks to find out this doesn't exist