Substance Mass Exporter (And Manager).... But it uses tons of RAM and won't clear! Help??

http://infinitypbr.com/files/SubstanceMassExporter.unitypackage

There's the project, please download it! It works, if you only do a few .sbsar files at once. It will...

* Batch Manage the "Generate All Outputs" options
* Display available textures and let you choose which to keep on export
* Allow you to save disk space/game memory by sharing unchanged textures on subsequent exports
* Toggle for setting exported normal map to "Normal Map" mode
* Toggle for creating a new standard shader material
* Will populate that material with the maps for you
* Toggle to convert the .tga files to .png

So it' does a lot.

Problem is, while Unity isn't really doing anything at all (literally, just adding things to the list), the RAM use goes up and won't come down. Many places in the script there's the following line, which should remove the RAM use for each material...but it doesn't.

Code (CSharp):

1. exporterObject.newMaterial.cacheSize = ProceduralCacheSize.None;
2. exporterObject.newMaterial.ClearCache();

Anyone know why this is happening? I plan on releasing this for FREE on the asset store, so any help you may be able to provide will help a lot of people who use substances.

 

For two substances, it looks like it's holding 250-300 MB RAM during the export process. Running the profiler (with Profile Editor set) shows the one frame of the export process (unity freezes during operation, so all stats are collected during that one frame).

On Deep Profile, you can see that most of the memory allocation is in ConvertToPNG(), specifically in Texture2D.GetPixels().



GetPixels is inherently very expensive in time and memory, so no real surprise here. Holding onto it after use, tho, is kinda strange. It might be a Unity bug or you may wish to play with setting a method variable array or a window level array to throw the pixels into and try to force cleanup on them.

Code (CSharp):

1.         Texture2D savedTexture = originalTexture as Texture2D;
2.         SetTextureImporterFormat (savedTexture, true);
3.         Texture2D newTexture = new Texture2D(savedTexture.width, savedTexture.height, TextureFormat.ARGB32, false);
4.  
5.         newTexture.SetPixels(0,0, savedTexture.width, savedTexture.height, savedTexture.GetPixels());
6.         //newTexture = FillInClear(newTexture);
7.         newTexture.Apply();
8.         byte[] bytes = newTexture.EncodeToPNG();
9.  

 

Unfortunately I'm actually getting the crash just by adding .sbsar files to the list -- well before I'm actually exporting anything. After loading unity, just bringing in a scriptable object that already has ~10 files attached to it will jack the ram usage up above 10GB.

 

I have theory on this but it's hard to prove. I suspect something was done in the latest versions of unity that has caused substances in particular to to consume ridiculous amounts of memory as well slow things down to a crawl. Unity is sometimes kicking off substance regeneration in places it did not in past versions. Worse still substances that did not change "seem" to be taking much longer to process than they did in the past (at the same resolution). It has gotten so bad that in some cases I want to remove all substances from the project all together and just use the exported images.

I do not know what "exactly" it is but there is definitely something wrong with unity's support of substances.

I'd be willing to bet when you say "Unity isn't really doing anything at all" that it is probably regenerating a substance in the background and consuming ridiculous amounts of ram while doing it.

Here are somethings I "suspect" are happening:
-simply looking at a single substance material seems to kickoff a regeneration of all material instances in entire substance archive
-Regenerating substances seems to take much longer and consume more memory than it did in the past (I need to get an older version of unity and time this versus newer versions)
-Changing things in the scene or doing things other than changing substance values (changing code??) seems to kickoff substance regeneration (I have yet to narrow down exactly what things are causeing full substance regeneration now)

Again I haven't fully tested these theories so I am hesitant about sending this to unity until I can say exactly what is wrong and prove it with a specific test.