Bug: 4GB limit to Textures in standalone build

There appears to be a hard limit of 4GB of textures in the standalone build. If your project has more than 4GB of textures, then some of the textures in standalone build will be messed up. While this limitation may be acceptable for mobile projects, this limit is completely unacceptable for PC projects.

There are currently two threads in the forum about this bug:
https://forum.unity3d.com/threads/textures-messed-up-in-windows-build.440675/
https://forum.unity3d.com/threads/texture-problem-graphical-corruptions-need-help.395985/

Can somebody from Unity please comment on this bug? Thanks.

 

Thank you for looking into this issue and commenting on this topic. As a short term workaround, I will look into using AssetBundles. I thought AssetBundles were for games that needed a way to download additional content from a server, which is a different use case from this.
https://docs.unity3d.com/Manual/AssetBundlesIntro.html

Big picture wise, there are several possible long term solutions for this issue:
1) Unity could add an optional resource file format that supports a 64bit unsigned integer for supporting projects with more than 4GB of resources. Unity could leave the existing 32 bit file solution in place for most use cases, and only use the 64 bit file format when the total resources exceeded 4GB. This would allow Unity to properly support large projects on PC without possibly breaking or slowing projects on mobile platforms. This could be similar to how Microsoft simultaneously offered multiple disk formats (FAT16, FAT32, and NTFS) during the transition.

2) Unity could implement splitting the large resources file into multiple smaller files using AssetBundles without forcing the developers to manually set up AssetBundles. This would offer a solution that provided support for large projects without breaking the existing file format.

3) Unity could add a pop up message in the Unity Editor during builds to let developers know their project is too big to fit into a 4GB resource file, and then include a link to a web page about the issue. At a minimum, Unity should immediately do this. Every one of us that ran into this problem has wasted literally days trying everything to fix the problem. Showing a detailed message at build time would have saved us all a lot of time.

What is not acceptable is letting developers run into a hidden limitation and then waste days of development trying to figure out why textures in large builds are suddenly broken. I get the concern about backwards compatibility, but that is not a good enough reason to let developers run into this problem. At a minimum, pop up a warning explaining the issue.

Also, leaving this 32 bit file size limitation in Unity is not really a viable long term solution. A lot of developers already think of Unity as a mobile-centric game engine. The only reason PC and console developers are not complaining more about this issue is because most people don't realize this limit/bug exists.

 

I am not using the Resources folder at all. I understand the Resources folder is not the proper solution. I link to my prefabs from my C# code through public GameObject variables and let Unity manage which resources get included in the build.

That is the frustrating part of this. I am expecting Unity to be able to manage the build properly. As a game developer, I should be able to expect Unity to handle the build properly. In this case, AssetBundles generates more work for me to handle something that Unity should be able to handle automatically.

 

It should just work fine with > 4GB if not using resources folder. Perhaps file a bug and link case here?

 

That is not my bug report. I submitted a bug report, but received an email that my bug report was a duplicate of that bug report.

My project is a single scene that I dynamically load everything into at runtime.

My Resources folder only contains an empty folder called Prefabs. There is nothing else in my Resources folder. Should I delete the Resources folder since I am not using it?

 

My scene file is only 13KB. The scene itself is mostly empty. The scene is just a handful of GameObjects that launch all of my prefabs dynamically.

For example, one of my GameObjects contains all of my object pool managers, which is small itself but loads thousands of projectiles into the scene. Similarly, I have another GameObject that manages all of the procedurally generated space scenes, which is another small object that launches a bunch of larger objects.

Are you saying the scene can only contain 4GB of static content, or are you saying the scene cannot dynamically load more than 4GB?

 

I am building a 64bit executable.

 

I did some additional experimenting with this problem today. I opened my backup from Friday, November 11. That is a point in my project before I purged enough textures to get the total build size under 4GB. Then I deleted the empty Resources folder, just to be sure that was not an issue. Then I built a Windows 64bit standalone of that project.

The messed up textures problem was there, just like in the following screenshot:
http://shilohgames.com/images/20161111/WindowsBuild.jpg

Total build size was 5.60GB.

The projectname_Data folder contains the following files:
app.info (1KB)
globalgamemanagers (179KB)
globalgamemanagers.assets (73KB)
level0 (1.91MB)
resources.assets (1.91MB)
resources.assets.resS (60.2MB)
sharedassets0.assets (834MB)
sharedassets0.assets.resS (4.54GB)
sharedassets0.resource (143MB)

 

Work arounds like that make it sound like Unity is only somewhat 64bit.

You have mentioned both AssetBundles and Multi-Scene as possible work arounds for this bug/limitation. I guess I will need to test both to see what helps.

 

I re-organized my project to use an AssetBundle. I am getting a crash when trying to use the AssetBundle that I generated. My AssetBundle is just over 3GB. What is the max size for an AssetBundle?

 

I gave up on AssetBundles, since there seems to be a limit on size of an AssetBundle. I refactored my project to use Multi Scene instead of AssetBundles, and that seems to be working. The next thing I will do is add a bunch of texture back into the project to push the total size up again to see if I can avoid the texture corruption bug as long as I do not reference more than 4GB of content per scene.

 

Multi Scene seems to be the ticket. I added a couple gigabytes of content to the scene, and retested. My total size was 5GB and I did not see any texture corruption in the Windows standalone build now.

What I did was move my SpaceSceneCreator GameObject to another scene called SpaceSceneAdditive. Then one of the classes in my main scene calls SceneManager.LoadScene (with LoadSceneMode.Additive) from its own Awake function, and then uses GameObject.Find from the Start function to locate that object.

My SpaceSceneCreator GameObject has a script that has references to about 3GB of content that I use to procedurally generate the background space scenes at runtime. I render the generated space scenes to a different camera, and then I bake that camera to a skybox cubemap. Then I use the 96MB material from that cubemap to use as the skybox on the main camera in the Main scene. So even though I have 3GB of space scene contents referenced, I only actually use the 96MB skybox material during actual gameplay. Anyway, this is all working great now.

Anyway, if anybody runs into a similar problem in their own projects with texture corruption on scenes with a lot of content, the Multi Scene option in Unity will work to get around the bug/limitation.

Now the big question is why that limit exists at all within Unity?

 

Thanks for your help with this, Karl. Multi scene is working great. I should be able to complete my current project using that as a work around.

 

I appreciate this thread as it helped get me past the same issue. I was considering moving to AssetBundles because of this, but AssetBundles seem like too much trouble for my project.

I found what I think is a nice solution that doesn't involve modifying much in existing setups or using the load merge functionality and find object on the merged scenes. Here is how it works:

1. keep your original >4GB resource scene with all the linked prefabs/assets/etc. intact. You won't need to modify it for this to work.
2. create a start scene with very few linked assets in it. Most games/apps have this anyway as a splash screen. Make this scene 0 in build list when building. Add an empty gameobject to scene and attach this script on that gameobject that loads your main scene at start:

Code (CSharp):

1.  
2. using UnityEngine;
3. using UnityEngine.SceneManagement;
4.  
5. public class SceneLoader : MonoBehaviour {
6.  
7.     public string sceneName;
8.     public bool loadOnStart;
9.     public bool loadAsync = true;
10.  
11.     public void LoadScene() {
12.         if (loadAsync) {
13.             SceneManager.LoadSceneAsync(sceneName, LoadSceneMode.Single);
14.         } else {
15.             SceneManager.LoadScene(sceneName, LoadSceneMode.Single);
16.         }
17.     }
18.    
19.     void Start () {
20.         if (loadOnStart) {
21.             LoadScene();
22.         }
23.     }
24.    
25. }
26.  

3. create a bunch of new scenes, one for each asset group you want. In these scenes, create an empty gameobject and attach this script to it:

Code (CSharp):

1.  
2. using UnityEngine;
3. public class PrefabHolder : MonoBehaviour {
4.  
5.     public Transform[] prefabsToHold;
6.  
7. }
8.  

4. Now link some of your prefabs on the gameobject prefabsToHold list. Save the scene. Make as many scenes as you want to break up your prefabs/assets. The PrefabHolder is your way of organizing your prefabs/assets into different buckets. You could extend/modify the script to work with other assets besides just Transform(prefab) if you like. The key to this working is the link to resource file.
5. When building project, include the start scene 1st, followed by all your asset scenes using the PrefabHolder, and lastly have your main scene. Order is important as Unity puts assets into resource file based on scene they appear 1st in. Later scenes can reuse the assets from earlier scenes.

You will find sharedassets0.resS contains start scene assets. sharedassets1...(N-1).resS contain the assets you linked in the PrefabHolder component for each scene. Lastly sharedassetsN will contain your main scene assets that were not linked in earlier scenes.

Multiple benefits to this approach:

1. fast loading start scene
2. partition of assets how you want into sharedassets1...(N-1) which is beneficial for incremental game updates to users
3. works around the 4GB limitation on .resS file.

ccs

 

The 4GB limit is an unprofessional kindergarden joke!
I'm expecting the unity devs should update the 32bit serialized file format to 64bit in unity 2017 in the first place.

Kind regards, Q

 

I agree. It was shocking to run into this silly limitation, and it was not obvious what the problem was when I ran into it. The textures break in the builds, but the Unity editor never tells what the real problem is. Having a hidden 32bit limitation in a 64bit build is crazy, especially for a product in 2016.

The multi-scene workaround is effective enough for a short term solution. I am still having good results using multi-scene.

 

wow.. if I only knew about this before.. Do you have any idea how much time I would have saved?
What a terrible system in place to cause such a nightmare for me, and months set backs due to a Unity bug?

Is there clearly stated ANYWHERE in the documentation?

I am seriously really disappointed in this, Sorry Unity - not everywhere who uses Unity is building mobile games.

 

I agree that this should be documented. Even if Unity had just put some information into the error log, it could have saved me a lot of time.

 

Just wasted 3 days trying to figure this out. Thank god I found this thread.

Is it so hard to add an error/warning when one of your asset files is above 4GB? Could of fixed the issue in 10 minutes if I had known. >.<

Sorry guys, not acceptable for a product that has cost me $1000s over the years.

 

Wait what?? This issue is still undocumented? Unity get your act together. At least document your god damn limitations for once. You do such perfect job trying to constatnly sell us your subsrciption services but having a mention about the 4gb build in a documentation seem like an impossible task.

I remember when i encountered this issue, i was lucky to get a solution in few hours since a friend pointed me to this thread. It took days for him to figure out this too.

 

Hi,

* did you know, that the limitation count on compressed textures only?
Or is the limitation count calculating all original files sizes are used as textures eg. a .psd with 1024x1024 and 25mb?

* And does all resourses folders are calculated into a big one?

@karl_jones
1) What is the difference between a resources data file and asset-bundle folder in build?
2) How can existing used references exceed 4gb because these data structure should be also go into any indexed file like asset-bundles and resource folder files?

3) Following example scenario:

A building that contains ~40 Rooms. Each of the rooms contain textures and meshes from another room as well, but 50% of each room has it's own unique data. Loading all data contains 16GB, so not possible to load all the stuff at once. Each room can be entered at any time.

For sure while entering a room the data should be loaded. But what is the preferred method to do in order to load the required stuff as fast as possible without hiccups and extra ordinary academic work.

1) Storing the rooms in one scene load/unload-async and let unity to handle all the references?

2) Using asset bundles, build a info structure contains what asset bundles to load/unload?
And is it the best way to asset-bundle.unload(true) because to hold all the material references in memory?

But wait...

If a bundle contain a texture must be loaded and the material already exist as loaded bundle, then the references doesn't work (as the documentation say), so I have to unload the asset-bundle.unload(false) with the existing material, load the asset-bundle containing the texture and reload the previous bundle with the corresponding material again?

But wait again..

What happen if there any references from new load bundles includes the Material are broken during last bundle.reload? In this case I must recursively search through all broken links and reload all existing bundles contain textures... then reloading these ones with materials again?

So if you suggesting me to use bundles, you must be assured and be able to explain exactly how to use these in this case.

Thank you..

 

If asset bundles are the recommended way to use for streaming too, why not going the logical way and build a folder name > AssetBundles in the project window.

Every additional child folder acts like a real single asset bundle itself. Further, managing the asset bundles can be done by the project manager as well, like the asset bundle browser tool does currently.

The AssetBundlesBrowser can rebuild as an AssetBundles::ServerFileManager where the AssetBundle-files have to be located or save/loaded from 1) http server 2) local host system 3) wlan network 4) etc.

Then the whole asset bundle system make sense in the workflow of unity.

 

We REALLY need you to make this change and allow >4gb in the resources file. Karl, you are suggesting using AssetBundles so I took off running with your idea... I was all excited for a few days, only to discover that loading my mobs from StreamingAssets takes WAY TOO LONG (60+ seconds to start my game), whereas when the same assets were in "resources.assets.resS", the game loaded in just a couple seconds.
I'm having to do something really unconventional (having as many as possible in resources, and the overflow as AssetBundles). This is not acceptable. As I add more to my game, it's going to take longer and longer to load, even in the editor.
P.S. loading just the asset I need at the time I need it was not an option. This is an MMORPG game and even loading one asset (a monster) took long enough that it could kill a player while its assets are loading.

Either need to make that pointer 64 bit (preferred option) or do something about the speed of AssetBundle loading. How can that huge 4gb resources.assets.resS file be loaded (or accessed) so quickly? Unity needs to do the same for external asset files.

 

This is really unprofessional and shortsighted of Unity to let this pointer-size issue just ride. Since when did we have 32-bit file systems with a 4gb file size limit? FAT32? How long ago was that? To add insult to injury, they ignore my questions. Well, if anyone is interested in knowing my progress, I discovered something that may help you. If you took Karl's suggestion and switched to using AssetBundles and you're experiencing very slow asset loading time, I found that you can save the assets as UNCOMPRESSED and they'll load much quicker. Of course, this takes up a lot more disk space but it works for now.

 

Is there an ETA on this change?

 

Given that Unity seems to be making a push to attract more cinematic quality projects(which would subsequently require larger and larger data files) with their tech demos, it seems to me odd that a user should have to jump through more hoops just to work around a limitation such as this one.

 

Here is a better question. Is there an ETA for documentation where you point out the limitation? More and more people are going to run on this issue and the answer is nowhere to be found. I have no problem with the issue that there are and will be technical limitations due backwards compatibility and etc.. (i am doing the workarounds pointed in this thread), but what i find unacceptable is that UT is unable to document this information properly.

This rant is not directed to you dev's. I understand that this is a large organisation and it is not your job to keep up the documentation. It is just very frustrating that UT finds a time to constantly spam me with their useless marketing fluff about services that do not help me in any way to make better games. But at the same time pointing out some very important issues are going unnoticed for years. People running on this issue are wasting days of their dev time until they accidentally find this thread by an abstract "my standalone build textures are messsed up" google search.

 

I just wasted the better part of a working week on this. So yes. A message would save us all a lot of time.

 

That is the exact problem with the current situation. When I ran into this problem, I literally wasted days trying to figure out what was wrong. Each person that ran into this problem would have loved a message in the Unity Editor to let us know what was going on.

 

If Unity generates a resource file larger than 4GB, the application will not work (properly).

If adding true 64bit resources support isn't simple, perhaps you can make the build fail instead (as a short-time solution)? That way we don't even have to run the game to figure out it isn't working and don't have to spend days trying to find the cause.

If Unity detects a resource file larger than 4GB while building, it could fail with an appropriate error message, like:

I added a feedback item for it:
https://feedback.unity3d.com/suggestions/fail-build-if-generated-ress-file-exceeds-4gb-file-size

 

This is a Silly issue if you consider things like SpatialOS and World Streamer. We now are building Huge worlds and this Limit needs to be removed. I just wasted 2 days on this issue until I found this post. And I question how Unity is storing Terrain Textures. Each should be loaded into memory once and reused. Since my scene has only a grid of terrains and their textures, I can tell you that it is not storing the 5 4K textures in an efficient manner...

 

I second making the build outright fail

 

How can I see if I have this issue with 4gb limit? I have the same screwd textures... this sucks Unity!!!!!!!

Is this the file in build ; sharedassets0.assets.resS ? its a littile over 4gb....

Does it count EVERYTHING thats in the project folder OR ONLY the textures you actually use in your scene? Can I make them smaller? they are 4k..

@karl_jones

 

Okay thanks! where can I find more info about the Asset Bundle? I found some free one on the Asset Store... The same one Github... Is it easy to use? Just drop textures there or ? ..

 

You probably meant 4GB there.

 

Thanks! one question tough.. I understand that theres need to be a script so it can load the assets..

var prefab = myLoadedAssetBundle.LoadAsset<GameObject>("MyObject");
Instantiate(prefab);

does that code line be for EVERY prefab and texture or ?

 

Okay thank you.

What if I have a gameobject in my scene that is a house (prefab) and I have furniture in the house and water bottle with script and food with script (pickup & use).. The house is the parent, furniture and food & water is children.. I drag the house (gameobject) to the asset bundle manager window and it goes there and I name it "House". Okay wohoo I have a asset bundle, I build and it creates the META and whatever....

I was thinking only to have houses etc as asset bundles...

BUT

When I build the game the default way, how does Unity know NOT to include the asset bundle asset & its belongings into the resource file?

Should I build the asset bundle asset to the same folder as the real game?

And the script that loads the asset should it be in the scene on some gameobject?

 

Isnt the asset you are using for the terrain reusing textures?