Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[5.2.x]Is Unity strips our shader when building and How Unity manage shader variants runtime

Discussion in 'General Graphics' started by Sparrowfc, Dec 23, 2015.

  1. Sparrowfc

    Sparrowfc

    Joined:
    Jan 31, 2013
    Posts:
    100
    First, it starts with a problem.

    I built streamed scenes for my game and some of the scene objects need to change from opaque to transparent during the runtime game play. So I wrote a standard like shader that can switch blend mode through setting shader paramters by script and with multiple variants such as '_NORMALMAP‘, '_ALPHABLEND_ON'. By using that shader, I can just write scripts to implement that result. The shader is stored under resources folder. It works in Editor but failed on device.

    After reading some documents, it occurs to me that maybe some shader varients is stripped automatically when build the project or those variants is not preloaded during the game play which eventually cause the failure. So as I look into the GraphicsSettings and see 'Always Included Shader' and 'Preloaded Shader' settings. After attach my shader into the always included shaders list, problem solved!



    Still I have bunch of questions about Unity shader variants mechanism, alone with assetbundle mechanism. I understand that in Unity 5.3, assetbundle no longer contains pre-compiled shader, so the problem may no exist. But since upgrading to 5.3 need to rebuilt all the assetbundles and rebacking all the scenes, our project may stay in 5.2 for a long time. So it's better to fix it out clearly.

    1. Adding 'Standard' shader into the 'Always included Shaders' list makes the application's runtime memory increase over 100MB. Sooooo scary, what happened? is that a bug?

    2. Creating a ShaderVariantCollection Asset and adding it to 'Preloaded Shader' list won't solve the problem. Maybe because I just can't add all the variants into the collection. It's a forwardbase shader and support lightmap and dirlightmap, so it has 'DIRECTIONAL', 'LIGHTMAP_ON/OFF', 'DIRLIGHTMAP_OFF/COMBINED', 'SHADOWS_SCREEN/OFF', '_FOG_LINEAR' keywords alone with several my custom keywords. The total variants count is over a hundred. It just too many variants to add in to the collection asset.

    3. Is that correct that adding a shader to the 'Always Included Shaders' list equals to creating a ShaderVariantCollectionasset which contains all the variants of that shader and adding that to 'Preloaded Shader' list.

    4. Some shader can switch varient runtime even if I didn't add it to the 'Always Included Shaders' list. That is which make me wonder if Unity automatically strip my shader variants at building time or automatically include some variants to preload at building time.

    5. If I call ShaderVariantCollection.WarmUp at runtime will those variants exits permanently, or I need to check every time I use those variants. Add still I'm trying to use the WarmUp API to replace the Always-Included-Shaders resolution, but yet not succeed.


    I'm gona build a sample project to digging those questions, and keep my newly found posted.

    Any suggestion or guidance will be really appreciated.
     
  2. Sparrowfc

    Sparrowfc

    Joined:
    Jan 31, 2013
    Posts:
    100
    OK, I think I've found the key of the problem : The Differences Between 'multi_comple' and 'shader_feature'. dam, just missing one really important note in the document 'the only difference is that unused variants of shader_feature shaders will not be included into game build'. http://docs.unity3d.com/Manual/SL-MultipleProgramVariants.html.

    So the actual reason that I can't change shader keywords runtime is because those shaders use 'shader_feature' but not 'multi_compile'!

    And when I drag those shader to 'Always Included Shaders' list, it force unity to build with all the variants of those shaders.

    And seems like multi-compiled shader can really takes up hundreds MB of memory. And maybe that's why Standard shader uses 'shader_feature' instead of 'multi_compile'.


    The Conclusion is just like what Unity documents says : Use multi_compile for shader features that you would like to change by script. It can also save lots of memory usage that add the shader to 'Always Included Shader' list
     
    Last edited: Dec 25, 2015
  3. Sparrowfc

    Sparrowfc

    Joined:
    Jan 31, 2013
    Posts:
    100
    And by the way, I have to say the multi_compile/shader_feature design really sucks. If I want to change shader keywords, I have to use multi_compile. However if I want to save runtime memory & package size, I need to use shader_feature for assetbundle to use. And when I need a shader to do both, the conflict is irreconcilable. The only way I can do it now is to change the shader code when I built the bundle and then make it back
     
  4. gfdgdfgdfg

    gfdgdfgdfg

    Joined:
    Sep 29, 2014
    Posts:
    7
    hi,i get a confuse about this "shader variants mechanism",i load assetbundle at runtime, this shader.parse cost too mush time. two assetbuble has same shader, but shader.parse execute two times, i think it should just execute at load first bundle.
    did you find this problem.
     
  5. Sparrowfc

    Sparrowfc

    Joined:
    Jan 31, 2013
    Posts:
    100
    yeah, Shader.parse takes most of the time of loading bundle. The best solution should be building the shader asset separately and letting other bundles depend on it, and load the shader bundle at begining. However creating dependency bundle would brings up other issues, eg. creating bundle from different project. currently I just leave this problem there. Instead I cache as many as resources loaded from bundle to lower the frequency of loading an assetbundle.

    For me now, it's the memory usage of those duplicated shader in different bundles matters. As I said, my shader is complicated and takes 0.5Mb in memory per-bundle. So imaging I combined a character with three parts: body, arm, foot, each part have 8 different type, 6 types of character in total. I would have 3 * 8 * 6 over a hundred bundle to load. And if all those bundles are loaded, the shader would takes over 50MB wasted RAM!
    And another thing I've found through memory profiler is that when loading those duplicated shader, others/ShaderLab increases a lot. Still working on it.
     
    Last edited: Dec 29, 2015
  6. gfdgdfgdfg

    gfdgdfgdfg

    Joined:
    Sep 29, 2014
    Posts:
    7
    i got the same problem like you. i just change from 5.1 to 5.3, not tested on 5.1, so i am not sure that is a 5.3 bug?
    and like you say "The best solution should be building the shader asset separately and letting other bundles depend on it, and load the shader bundle at begining". i had do like this, bundle A B C depend on D, D is the shader bundle. i load bundle D first, then i load A,B,C, the shader.parse execute 3 times, that really confuse me.
    ps , the D bundle just have the build in vertexlite shader. it's a build in shader, why it need to reparse
     
  7. gfdgdfgdfg

    gfdgdfgdfg

    Joined:
    Sep 29, 2014
    Posts:
    7
    you said "Instead I cache as many as resources loaded from bundle to lower the frequency of loading an assetbundle."

    i do this before, got a big memory use like you said below, so i try to unload bundle when not need, and reload it when use, this make memory reduce. bug got shader.parse problem when load bundle.

    i think the best solution is to fix the shader.parse problem. Still working on it.
     
  8. Sparrowfc

    Sparrowfc

    Joined:
    Jan 31, 2013
    Posts:
    100
    Cache resource is different from cache bundle, it's a bit complex but worth trying. say if you want to instantiate an object in bundle, you do following steps

    1. try get it in object cache, if found goto 6.
    2. (cache not found) load assetbundle and get ResObject through bundle.LoadAsset(Async)()
    3. cache ResObject / hold reference
    4. call bundle.unload(false) then Resource.UnloadUnusedAsset()
    5. back to step 1.
    6. instantiate the ResObject get ObjectX and add a reference count to ResObject
    7. when ObjectX is destroyed remove the reference count of ResObject
    8. when ResObject's reference count is 0 for some seconds, remove ResObject from cache and call Resources.UnloadUnusedAsset()

    That will release bundle memory but hold resource memory. Remember even if you don't hold ResObject reference, it still takes up memory space until all the instances has been destroyed.
     
  9. Sparrowfc

    Sparrowfc

    Joined:
    Jan 31, 2013
    Posts:
    100
    Shader in 5.3 is different from 5.2, I can't say anything about that. This will affect AssetBundles built in previous versions of Unity - they have compiled shader assets inside them by definition. Any shaders in such bundles will need to be rebuilt. http://docs.unity3d.com/Manual/UpgradeGuide53.html
     
  10. gfdgdfgdfg

    gfdgdfgdfg

    Joined:
    Sep 29, 2014
    Posts:
    7
    may be this cause Shader.parse problem. i will check on 5.1 to chcek this
     
  11. gfdgdfgdfg

    gfdgdfgdfg

    Joined:
    Sep 29, 2014
    Posts:
    7
    my project can not use this, memory is a reason, but the key point is that i need to run in a big map, can not load all npc at start, must load them when player can see them, so we need to load when player runing, load cause Shader.parse, that casue hiccup in main thead