1. We've introduced thread tags, search within a thread and similar thread search. Read more here.
    Dismiss Notice
  2. We've released our first Timeline Experimental Preview, our new tool for creating cutscenes and more! To check it out click here.
    Dismiss Notice
  3. All Unity Pro perpetual license customers: your special offer to subscribe is ready! Click here.
    Dismiss Notice
  4. Unity 5.5 is now released.
    Dismiss Notice
  5. Check out all the fixes for 5.5 in patch releases 1 & 2.
    Dismiss Notice
  6. Unity 5.6 beta is now available for download.
    Dismiss Notice
  7. Get prepared for the Tizen Mobile App Incentive Program! Read more about the upcoming program here.
    Dismiss Notice
  8. Enter the Microsoft Developer Challenge for a chance to win prizes. Read more about it here.
    Dismiss Notice

[SOLVED] 64 Keyword Limit questions

Discussion in 'Shaders' started by n00body, Jun 20, 2014.

  1. n00body

    n00body

    Joined:
    Dec 17, 2010
    Messages:
    607
    So I found out that we are limited to 64 keywords via Unity's shader documentation, and naturally had a few questions:
    1. How does the system count the number of keywords? Does it include ones that are only ever defined within a shader and not used outside it? Is it limited to those defined as part of #multi_compile directives?
    2. The documentation says it applies to all shaders in a given project. So does that mean I can reduce the count by removing shaders I am not using?
    3. Is there any way that I can see the current keyword total for my project?
    4. Why does Unity even have this seemingly arbitrary keyword limit? Is it some kind of hardware limitation? Is it a legacy thing?
    5. Is there a limit on the number of #multi_compile directives a shader can have, or is the keyword thing the only real limiting factor?
    Thanks for any help you can provide.
     
  2. Rld_

    Rld_

    Joined:
    Aug 21, 2012
    Messages:
    140
    No one probably knows everything for sure (unless it has been posted somewhere), as it is likely handled within the unity shader compiler and unity itself, but for what I would assume:

    1. I don't know how the keywords are counted. I can only assume this is being done when the compiler does its work and keeps track of it. Not sure what you mean with defined outside of the shader, but considering the doc talks about shader variants, I think it's safe to assume it only counts for the mulit_compile directives.
    2. Likely, it depends if the count is being done for the shaders that are being used during play or in the editor.
    3. Not that I know, but you can always count by hand :p
    4. Perhaps the compiler starts to have trouble when going over more than 64. Who knows.
    5. Only one way to find out, not sure if you want to have 64 variants in one shader though.
    Not that much help I'm affraid, but it's something :)
     
  3. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    This counts all the keywords ever seen, in the whole project. Basically, each keyword is assigned a number, and there can only be 64 of them. In all shaders in the whole project that were ever loaded.

    FYI, Unity 5.0 will very likely increase the limit to 128.

    Yes.

    No, not really :|

    We use a bitmask to select the "most appropriate shader variant", and having up to 64 keywords in total allows that bitmask to be conveniently represented as a 64 bit integer. 64 bit integer operations are reasonably fast.

    No, the maximum number of keywords is the only limitation.

    For Unity 5.0, we're planning to make it easier to not "use up" two keywords for simple on/off features. E.g. if you want to do a "#pragma multi_compile FOO_OFF FOO_ON", then it should be possible to represent this with just a single keyword - if it is not set, then it's off; otherwise it's on. So in 5.0 we'll have a multi_compile syntax to enable that.
     
    Arkade and n00body like this.
  4. metaleap

    metaleap

    Joined:
    Oct 3, 2012
    Messages:
    589


    Wait, this is two things. There's "all shaders in the whole project" and "all shaders in the whole project that were ever loaded".

    I'm using only a handful .shader files in the final output, but like to have 100s of imported assets (sans their scripts) "in the project" for quick & easy reference, 99.5% of those don't ever make it into the build. (But how else to instantly access any assets I own during prototyping and technique-researching without even alt-tabbing, let alone lengthy unpack-and-import procedures all the time.) So anything that isn't referenced in any Scene or is in a Resources folder or in the "always included shaders" list shouldn't be counted "part of the project" IMHO, especially as far as shaders are concerned.

    Since I imported Skyshop yesterday, I keep getting the "Maximum number (64) of shader keywords exceeded, keyword AZ_EMIT_REFL will be ignored", this keyword isn't known to me, isn't used in any shaders that are ever loaded by my scene, 100% guaranteed. I can still enter Play Mode but I suppose next time I wanna Build this error will bite me.

    I considered a custom import postprocessor renaming .shader files into .shader.txt but what a messy hack :D
     
  5. n00body

    n00body

    Joined:
    Dec 17, 2010
    Messages:
    607
    @Aras
    I'm guessing this system seemed like a real elegant solution to this problem until people like me came along and started writing "kitchen sink" shader sets that abuse multi_compile. :p

    I'm not exactly thrilled by these answers, as I plan to add many more features to my shader set to make them more robust. I guess I will have to find ways to avoid using keywords.

    PS.
    Code (CSharp):
    1. #pragma multi_compile _THING_TAG_OFF _THING_TAG_ON
    2.  
    3. #ifdef _THING_TAG_ON
    4.     #define _LEGACY_THING_TAG
    5. #endif
    Does Unity count this as three keywords?
     
  6. metaleap

    metaleap

    Joined:
    Oct 3, 2012
    Messages:
    589


    Only 2, if custom #defines were counted as keywords, I'd have killed myself already :D

    I've taken to reducing some on/off keywords to uniforms in select cases, especially since pre 4.5 compilation times were so excessively long and knowing that static branching (ie. on uniform/constant values) isn't a significant cost anymore whatsoever on basically all current platforms.
     
    n00body likes this.
  7. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    Correct, that case is two keywords.

    And yeah, using uniform-based branching is a very practical approach. That is effectively "close to being free" on most modern hardware.
     
    n00body likes this.
  8. rea

    rea

    Joined:
    Oct 10, 2009
    Messages:
    1,726
    Just curious, what happen if there's no keyword limit?? (Honestly i don't know about this, that's why i'm asking :D)
     
  9. metaleap

    metaleap

    Joined:
    Oct 3, 2012
    Messages:
    589
    I'm still on 4.3.4 but (correct me if I'm wrong) as far as I can tell this is already well supported: if one just does this:

    Code (csharp):
    1.  
    2. #pragma multi_compile MYTHANG
    3.  
    Then one can EnableKeyword("MYTHANG") and DisableKeyword("MYTHANG") -- voila, an on/off toggle with a single keyword!

    Seems to work well.. am I missing some hidden lurking danger here or what? :D
     
  10. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    @metaleap, no, that should not work. What this basically does, is always define MYTHANG in all shader variants.
     
    n00body likes this.
  11. metaleap

    metaleap

    Joined:
    Oct 3, 2012
    Messages:
    589
    Then why the heck does it work so incredibly and reliably and consistently well right now for me? :D

    I tell you, I use it with #ifdef MYTHANG and it's indeed off when I DisableKeyword and back on when I EnableKeyword..

    Unity's mysterious ways :p probably it'll stop working like it does right now next time I restart Unity or something. Well good to know it "shouldn't" work, then I won't go crazy when it stops working....
     
  12. n00body

    n00body

    Joined:
    Dec 17, 2010
    Messages:
    607
    @Aras
    Just to confirm, the keyword count only increases with multi_compile declarations in shaders, correct? I just want to make sure it is not affected by parameter definitions that use dropdowns and toggles. Those controls would still be useful to me as float values.
     
  13. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    It is affected by anything that causes a "shader keyword" to be created. That is mostly multi_compile, but some MaterialPropertyDrawers (e.g. KeyworkEnum) also create keywords.
     
    n00body likes this.
  14. n00body

    n00body

    Joined:
    Dec 17, 2010
    Messages:
    607
    I guess I should just avoid Toggle and KeywordEnum for now. The only place where I might need a toggle, I can do just as easily with a slider.

    Thanks for the help everyone.
     
  15. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    For Toggle, you could make your own that only sets the float to 0..1 (but does nothing about keywords). For enums, there's already [Enum(name1,value1,name2,value2,...)] if you don't need keywords.
     
  16. cician

    cician

    Joined:
    Dec 10, 2012
    Messages:
    227
    I'm sorry to say it, but this limitation makes the whole multi_compile thing pretty useless, in particular for an asset store vendor.
    I'd exchange happily the global 64 keyword limit for 1 keyword per shader.
    Most people already have something like skyshop using most keywords already and things will break randomly based on order of packages being imported.
    I was all hyped up for the combination of Toggle and fast shader compilation in 4.5, but I realized that I just can't realistically take advantage of it. :(
     
    ksam2 likes this.
  17. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    Yes. That's why we're making the limit higher in Unity 5.0 (to 128). And yeah, a proper solution would be to have some limit of "global keywords" (e.g. 32) and some limit that is per-shader (e.g. 32). But that is quite a bit involved to implement, so 5.0 will have a "stopgap" solution of increasing the limit a bit.
     
  18. yezzer

    yezzer

    Joined:
    Oct 30, 2009
    Messages:
    123
    I'm having the "64 Keyword Limit" problem when using a bunch of 3rd party assets in one project (Alloy, Skyshop, Amplify Color, TC Particles and several others).

    I've removed a lot of shaders I won't use, but am still having this issue. Is there any kind of fix, or can I see what shaders are using which keywords, etc?
     
  19. Chman

    Chman

    Unity Technologies

    Joined:
    Feb 6, 2010
    Messages:
    684
    Yep, you can select the shader in Project and click the Show button in the Inspector (Variants line) or you can open the shader and look for #pragma multi_compile to see what's in use. But if you want to remove some keywords you'll have to modify the shaders (and probably the material editors as well, if any)...
     
  20. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Messages:
    924
    Today I switched from 3.5.3 to 3.5.5 with the same project and now I have that error messages, too.
     
  21. Darkcoder

    Darkcoder

    Joined:
    Apr 13, 2011
    Messages:
    989
    Wow, I just came across this issue myself. I agree with cician, this limitation is a massive issue for asset developers that write shaders. 64 or 128 across the whole project, wtf? There absolutely needs to be a local keyword list, how often would you need these to be global? I guess I'll have to rename mine to A, B, C,D etc to minimize waste :/
     
  22. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    Yeah I know. The problem is that with current API/shadersyntax, it's impossible to know which keywords are "local" (to the material), and which ones are "global". You can write #pragma multi_compile FOO BAR and then either do Shader.EnableKeyword("FOO") or material.EnableKeyword("FOO"), and we don't know which one's going to happen beforehand.

    So for 5.0, interim workaround is to raise the limit to 128.

    For a "proper fix", we'd have to do some breaking changes at some point, where you'd need to somehow indicate which keywords will be global vs local. Suggestions on how to do that?
     
  23. Darkcoder

    Darkcoder

    Joined:
    Apr 13, 2011
    Messages:
    989
    #pragma local_multi_compile ?
     
  24. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    Right. And then if some shader specifies "FOO" as a local keyword, but another shader specifies it as a global one; then "something" should happen that would hopefully either make both of them still work, or a nice error would be printed somewhere.
     
  25. Darkcoder

    Darkcoder

    Joined:
    Apr 13, 2011
    Messages:
    989
    Indeed, some sort of redefinition error would work if it's easier to implement. Though if the shader author is defining a local keyword I think it would be assumed they're not using the global version, or in the case of assets not aware of the other version, so it would make sense for it to override it.

    In any case, I've now renamed all my keywords and prefixed them, so it should cut down on clutter and prevent any clashes.
     
  26. Razieln64

    Razieln64

    Joined:
    May 3, 2008
    Messages:
    122
    Hopefully Unity 5 will be released soon. I already cleaned up my shader keywords as much as I could and when I got SGT imported in my project it broke again. :(
     
  27. Darkcoder

    Darkcoder

    Joined:
    Apr 13, 2011
    Messages:
    989
    SGT only uses 10 keywords, so you should have 50 or so left to play with. If the keywords are from your own shaders then try sharing the keywords across as many shaders as you can.
     
  28. lxpk

    lxpk

    Joined:
    Feb 3, 2009
    Messages:
    34
    I've run into the 128 shader keyword limit in Unity 5 by using UBER, TextMesh Pro and other shaders. All the text in my game has stopped rendering. This thread says "SOLVED". What is the solution to manage shader keywords? If the 128 keyword limit is what we are stuck with for now, where is the shader keyword audit list that helps us count the keywords and plan which shaders to remove? This is seems like an undocumented (mentioned at the bottom of an article once) limitation that breaks a project with no solution path offered other than to start gutting your project randomly hoping it goes away. Am I missing some guidance on how to fix or manage this? It seems like not very many Unity projects use all the high-end shaders often enough for this problem to get as much attention as some others.

    This issue could use some votes or some feedback as to how to fix the problem in its current form:
    https://feedback.unity3d.com/suggestions/remove-global-limit-on-number-of-shader-keywords

    Otherwise the current solution for shader-heavy projects is this:
    https://forums.unrealengine.com/showthread.php?89085-what-is-the-Maximum-number-of-shader-keywords

    Here are multiple threads with people struggling with this and resorting to manually inspecting .mat files with text editors looking to fix Unity's unauditable shader keyword reservations.

    http://forum.unity3d.com/threads/maximum-number-128-of-shader-keywords-exceeded.320307/
    http://forum.unity3d.com/threads/max-shader-keywords-exceeded-solution.225586/
     
    Last edited: Apr 22, 2016
  29. pointcache

    pointcache

    Joined:
    Sep 22, 2012
    Messages:
    283
     
  30. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    Currently your best bet is to remove shader keywords from the shaders/materials that you don't use, indeed.

    When the limit is exceeded, all the keywords that are present are printed. I'd say two things then:
    • Search for the keywords the shaders of your project, see which ones are needed vs which ones are not. Try to remove/reduce (or just delete the shaders if you don't need them?)
    • Similarly, if you have keywords serialized in material assets, but don't need them, then writing up some editor scripts that remove them is one solution, or altering the serialized material files directly.
    We are working on improving the situation, but it's hard to do that without breaking some possible corner-case usages.

    The current plan is to have separate "global keywords" (the ones set from Unity itself, or from your script code etc., generally the "multi_compile" variery) and "local to this shader/material only" keywords (generally of "shader_feature" variety). And there, for example, would be up to 64 global ones and up to 64 local ones, or something like that.
     
    n00body likes this.
  31. ksam2

    ksam2

    Joined:
    Apr 28, 2012
    Messages:
    831
    After what Aras said about removing some keywords I start to creating a new project and importing all my stuff and assets again, I was removing lots of shader and assets, creating a real clean project but still only with 10 assets that use keywords I got 128 Limitation Error and I figure it out that is not possible to have enough assets for a big project or even small games.

    I was never disappointed Unity that much. it's the biggest unity limitation
     
  32. cician

    cician

    Joined:
    Dec 10, 2012
    Messages:
    227
    I take this opportunity and blame Unity Technologies for lack of new features in my shader :p

    Seriously speaking, my asset would greatly benefit from the feature, but I cannot possibly publish a shader using it to the Asset Store because the support would be pure nightmare. I'm always reluctant of adding new features because of shader variant explosion. Does the user want detail textures? multiple specular lobes? ... tessellation?
    Another solution would be a material editor, like Unreal's (or Shader Forge). Just sayin'.
     
  33. rea

    rea

    Joined:
    Oct 10, 2009
    Messages:
    1,726
    I'm gonna ask this question again since nobody answering. . . .

    So @Aras why?
     
  34. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    It's not the number of assets, it's number of shader keywords used. If you have a single shader that tries to expand to 2 to power of 128 variants (that's a lot of variants), you will hit the 128 keywords limit right there.

    Do you have a list of assets/shaders that are the most affected, and the list of keywords they use? (searching through *.shader files for #pragma multi_compile or #pragma shader_feature would answer the "which keywords they use" question)
     
  35. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    We could remove the limit, at expense of making all rendering slower.
     
  36. ksam2

    ksam2

    Joined:
    Apr 28, 2012
    Messages:
    831
    I know, but I need to use most of Uber shaders or RTP features. Can this number increase to 256 at least?
     
  37. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Messages:
    18,447
    No thanks.
     
  38. ksam2

    ksam2

    Joined:
    Apr 28, 2012
    Messages:
    831
    Actually this limitation should be remove or atleast whit more keywords limitation like 256 (If this is possible).
    I'm stuck on my project somehow because of this.
     
  39. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Messages:
    18,447
    Yeah but that's your (solvable) problem. If unity craps all over the speed to fix that, it becomes an unsolvable problem (lack of speed). A slower Unity means a Unity nobody really wants.

    What I think Unity needs to do is make more diagnostics for shaders, or just rethink this process a bit. It could be your materials are causing this shader keyword limit to be hit, even if they're no longer using those options.
     
    n00body likes this.
  40. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Messages:
    2,812
    I have hit this problem repeatedly and Unity really needs to either A) Fix it or B) at least stop making the problem worse by using up all the shader keywords for their own stuff. Just a basic Unity install with some SpeedTrees, the Standard Assets, and the Image Effects seems to use up much more than half of the keywords, which means there's hardly any room left for other assets. I wanted to use the new DoF from the Unity Cinematic Effects but LOL NOPE, of course I can't, it has a million shader keywords in it. :(
     
    radimoto, ksam2 and AcidArrow like this.
  41. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Messages:
    2,812
    If there were a toggle to turn the limit on or off I would gladly use it even if it cut my framerate in half, because then at least I could develop without needing to write batch scripts to search through material yaml files for keywords anytime someone adds something to a project. I could put off the constant keyword juggling until the project is ready to release.
     
  42. ksam2

    ksam2

    Joined:
    Apr 28, 2012
    Messages:
    831
    No one listening, can we talk to the Unity boss?
     
    Last edited: May 15, 2016
  43. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    We are listening, and solving the root cause of the problem in the background.

    But meanwhile, people who are running into the issues: can you list the packages + extensions that you use? I just checked for example having whole ReliefTerrainPack + UBER Shaders + Speedtree being used, and in 5.3.4 that still leaves 40 shader keywords to spare.
     
    n00body likes this.
  44. rea

    rea

    Joined:
    Oct 10, 2009
    Messages:
    1,726
    okay fair enough. . .let's not then
     
  45. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Messages:
    2,812
    I'm using RTP, Uber, SpeedTree, Standard Assets, Time of Day, and either SCION or PRISM for full screen effects. I think that put me over the limit until I manually edited ToD and Prism shaders. However, to me, and I'm sure a lot of other people, being told to just limit our assets to three or four big things is a pain even if I don't actually need fifty assets for release. For example, I like having the Level 11 subscription and whenever there is free stuff or something I pick up on sale, like some cool explosion effects or something, I like to import it into my own project to see how it looks alongside everything else rather than just relying on creating a separate empty project and playing with it. But the shader limit means you generally can't do that. My situation with the new Unity Cinematic package is a perfect example: I'd bet almost all the people trying it out want to be able to just import it into their main project to see how it looks rather than starting a new blank one, trying to figure out what effects they like one by one, manually editing the shaders, and carefully copying things over manually. The whole point of asset packages and the asset store is to avoid that process.

    I don't know what solution you guys are working on but it seems like the best thing to do would be to introduce some sort of "namespace" for shaders. Each shader will belong to one namespace which has 64 keywords and access to the global namespace which also has 64 keywords. That way shaders within a package can all share the same namespace but won't interfere with other things. Of course, this would mean Unity would need to go through some of their own assets and move things into a namespace, since I think currently you use up more than 64 before even adding anything from the asset store.
     
  46. ksam2

    ksam2

    Joined:
    Apr 28, 2012
    Messages:
    831
    Here is my packages.

    Standard Assets
    azure[Sky]
    Colorful
    FilmicTonemappingDeluxe
    ReliefPack
    ScionPostProcess
    Natural Bloom
    SSAO Pro
    Sunshine
    TextMesh Pro
    UBER
    Cinematic Effects
    Volumertic Light-master


    and some other image effects that uses 7-8 keywords,

    I start to create a new project and import all those asset again with a new scene but I have keywords limitation error again!
     
  47. Aras

    Aras

    Graphics Plumber Unity Technologies

    Joined:
    Nov 7, 2005
    Messages:
    4,521
    Ok thanks for the info. So here's what we're doing:

    1) For "soon" (as in, doing right now, will try to get into 5.4), I'm cleaning up some of our own shader keyword usage. Looks like I'll be able to save something like 5 shader keywords in engine + built-in shaders. People doing cinematic image effects will likely be able to salvage a couple keywords in their shaders too.

    2) For "actual solution" (definitely not in 5.4 though), we are working on solving the root cause of the problem. Similar to the "namespacing" idea mentioned above (some keywords would be "global", some "local to the material" etc.). This is fairly involved, and not finished yet at the moment.

    In the meantime, your best bet is to not put too much of unused things into the project. For example, from one of the list above - it's quite likely that you don't need both the cinematic effects Bloom and Natural Bloom at once (same for SSAO Pro & cinematic effects SSAO etc.). Yes I understand that it's far less than ideal...
     
  48. Chman

    Chman

    Unity Technologies

    Joined:
    Feb 6, 2010
    Messages:
    684
    For what it's worth we just removed 3 shader keywords from the Cinematic Image Effects package (on Bitbucket only at the moment). We'll probably remove some more in a near future.
     
    n00body, ksam2, iMP and 1 other person like this.
  49. jack-riddell

    jack-riddell

    Joined:
    Feb 11, 2014
    Messages:
    301
    A lot of the features in my shader don't need to be dynamic at run time.
    Things like emissive settings or shadowing style are decided at edit time and can not be changed when the game is running due to different art asset requirements.
    I could create 2 versions of my shader (one with emissive and one without) and use a smart material editor to switch out shader versions in the background but if my shader has 5 optional settings that's 32 shader versions to maintain.
    At the moment I am using #Shader_Feature to organize most of my shader variations but that adds to the total shader Key Word count.
    Why cant you preprocess the #Shader_Features compiling them out into the shader code they define and save all the key words for the run time stuff?
    From what I understand the #Multicompile stuff is for things that change at run time like lighting setup or fog settings it seems strange to build a system that bundles static shader settings in with dynamic ones.
     
    makeshiftwings likes this.
  50. ksam2

    ksam2

    Joined:
    Apr 28, 2012
    Messages:
    831
    I've deleted two other of my assets to get ride of this error for now! (Sunshine AND Cinematic Effects removed)
    I hope Aras save us and solve this limitation somehow. still I need to add more assets.

    Can anybody create a keywords counter asset? something that show us the number of all keywords used in project.
    This can be useful for now!
     
    lxpk likes this.