Search Unity

Unity hangs making build during Light Transport

Discussion in 'Global Illumination' started by radiantboy, Apr 24, 2017.

  1. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,633
    If I leave real time global illumination on and try to make a build, it always get stuck during light transport jobs, in the end I have to kill unity with the task manager. Anyone know how to fix this?
     
  2. Glockenbeat

    Glockenbeat

    Joined:
    Apr 24, 2012
    Posts:
    670
    Same here. I have a fairly big scene (using realtime GI only) which always takes a while to build, but witin the editor it always finishes the GI steps. However once I am trying to build the game it either gets stuck in 5/11 clustering or 7/11 Light Transport (at the time of writing with 47 jobs left without any progress), although all steps have already been completed before via auto generate.

    I've tried various things, replaced a fair amount of assets within the scene, even tried to reduce the complexity for testing just to find "that one" object which might cause trouble, lowered GI settings and various object lightmap scales but no luck.

    There's also no output in the logs indicating any error. CPU stays at roundabout 10% load (4790k), memory usage at roughly 1 GB (out of 16 GB totally available) without any real changes, even if it's left "active" for several hours.

    There are no differences in target platforms, also I can reproduce this on both a Windows Desktop as well as a Macbook Pro.

    The weird thing is: It worked before. The game is already published to stores and I am working on an update, however out of a sudden it stopped working, no matter which version of Unity I am using (tried several versions of 5.5 through 5.6).

    So, bottom line is - I'm unable to build the game. Any help is much appreciated!
     
  3. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,633
    Sounds identical to my issue, I wish it would bundle up what has already been generated, but it doesn't. Ive moved to Unity 2017 to see if this is fixed, just fighting against shader errors stopping me make a build and then I will try again and see what happens.
     
  4. Glockenbeat

    Glockenbeat

    Joined:
    Apr 24, 2012
    Posts:
    670
    Have you been able to achieve any progress? No progress from my side, tried several things, but didn't find any fix so far...
     
  5. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,633
    Nothing yet I have turned off the GI for now ;-/
     
  6. Glockenbeat

    Glockenbeat

    Joined:
    Apr 24, 2012
    Posts:
    670
    I actually found a solution!

    Do not rely on the GI cache and "Auto Generate" option. It seems like Unity has some serious issues loading the lighting data from the GI cache and somehow stops once it should load certain data, causing the build to stuck infinitely. Actually I consistently observed Unity hanging during clustering, light transport or even during reflection probes, no matter what I tried.

    The solution is to turn off "Auto Generate" and hit "Calculate Lighting" (or bake reflection probes from the dropdown) once you think you need a different light calculation. Unity will then go through all the steps and persist all the reflection probes and lighting data in a folder with the same name as the scene right next to the scene. From there it loads it during builds without any issue. Applies to realtime as well as baked GI.

    Another good thing is that you can actually add all these files to your VCS and have the same data being used across machines and platforms - for example my usual setup is to work on a Windows PC and produce windows builds, but switch to Mac once in a while. This way the other machine does not need to calculate and store the same data all over again just to populate its GI cache, failing at the very same build steps.

    This completely solved my issues, I now just need to remember to manually calculate the lighting once needed. Hope this helps you too!
     
    HSochnik and Dwarf33 like this.
  7. Lost-in-the-Garden

    Lost-in-the-Garden

    Joined:
    Nov 18, 2015
    Posts:
    176
    We are having the same issue here (currently on 5.6.0p4, everything was fine in 5.5.2p2), getting stuck in the Visibility phase. We'd prefer to use auto generate with a GI cache that resides on our NAS, instead of having to manage GI manually. One of the advantages here is that we can do nighly builds that generate all the GI for all levels in one go.

    edit: Just tried generating the lighting data manually, and the resulting LightingAsset has 130MB. We probably don't want to check that into our git... O:
     
    Last edited: May 10, 2017
  8. Lost-in-the-Garden

    Lost-in-the-Garden

    Joined:
    Nov 18, 2015
    Posts:
    176
    I just disabled auto update for all scenes, cleared the GI from all scenes, set up a new cache folder on our NAS and, enabled auto update for one scene and tried a build: Same result as before, but I guess we can rule out broken/stale cache data at this point.

    Would love to hear the thoughts from someone at @Unity
     
  9. Glockenbeat

    Glockenbeat

    Joined:
    Apr 24, 2012
    Posts:
    670
    I just did a quick search: A custom build script (static method) could actually perform the build for you during nightly builds. During that build (or better: before) it seems like you are able to calculate the lightning stuff via Lightmapping.Bake() or Lightmapping.BakeAsync().
    https://docs.unity3d.com/ScriptReference/Lightmapping.html

    You can also call that method from the command line via -executeMethod <ClassName.MethodName>
    https://docs.unity3d.com/Manual/CommandLineArguments.html

    Haven't tested it, but I can image it might work.

    However, I definitely also prefer a working built-in solution rather than doing that stuff manually. For me personally it is ok to do that right now (even checking in the lighting data) and I definitely prefer that over "no build at all", but Unity folks need to have a look at that.
     
    Lost-in-the-Garden likes this.
  10. Lost-in-the-Garden

    Lost-in-the-Garden

    Joined:
    Nov 18, 2015
    Posts:
    176
    We already have a custom build pipeline so integrating this would be fairly straightforward. There are a couple of other 5.6 issues that make the decision hard for us to take this step, but ideally we want to be on the latest unity version before entering cert.

    It's aways a waiting game, so see if and when the issues are fixed.
     
  11. Lost-in-the-Garden

    Lost-in-the-Garden

    Joined:
    Nov 18, 2015
    Posts:
    176
    Until these issues are resolved, we'll stick with 5.5. Not the thing I like, but we don't have much more time to sink into this. Here's the last thing I tried, which for some reason did not work:

    The approach was to leave the "auto" setting enabled in the editor, and then during the build, change all scenes to manual building, build the GI for each scene and then finally start the build method so that it will hopefully use the lighting Data asset.

    Unfortunately, starting the GI build from script, resulted in nothing, just a call that never returns and that seemingly does nothing else, looking like a spinlock on one thread. I also tried Async Building, but the progress never was more than 0 and it never completed, even though the GI solution should be valid in the cache.

    Here's the code, I hope that it helps someone further. Just call the functions before and after the actual build command. In you solution you would have to replace everything "TrackList" with whatever construct you have that hold the scenes that have to be built.

    Code (CSharp):
    1. private static bool ApplyGIHotfix(TrackListConfigData trackConfig)
    2.         {
    3.             // GI hotfix
    4.             for (var i = 0; i < trackConfig.TrackList.Count; i++)
    5.             {
    6.                 var track = trackConfig.TrackList[i];
    7.                 EditorUtility.DisplayProgressBar("GI Hotfix", "Building GI for " + track.name, ((float)i / trackConfig.TrackList.Count));
    8.                 EditorSceneManager.OpenScene(track.Path);
    9.                 Lightmapping.giWorkflowMode = Lightmapping.GIWorkflowMode.OnDemand;
    10.                 Lightmapping.Bake();
    11.                 //Lightmapping.BakeAsync();
    12.  
    13.                 //var spinnerArr = new string[] { "-", "\\", "|", "/", "-" };
    14.  
    15.                 //while (Lightmapping.isRunning)
    16.                 //{
    17.  
    18.                 //    var spinner = spinnerArr[System.Convert.ToInt32(EditorApplication.timeSinceStartup * 4.0) % spinnerArr.Length];
    19.                 //    if (EditorUtility.DisplayCancelableProgressBar("GI Hotfix", "Building GI for " + track.name + "... " + spinner, Lightmapping.buildProgress))
    20.                 //    {
    21.                 //        Lightmapping.Cancel();
    22.                 //        EditorUtility.ClearProgressBar();
    23.                 //        return false;
    24.                 //    }
    25.                 //}
    26.                 EditorUtility.DisplayProgressBar("GI Hotfix", "Saving " + track.name, ((float)i / trackConfig.TrackList.Count));
    27.                 EditorSceneManager.SaveOpenScenes();
    28.             }
    29.  
    30.             EditorUtility.ClearProgressBar();
    31.             return true;
    32.         }
    33.  
    34.         private static void ReverseGIHotfix(TrackListConfigData trackConfig)
    35.         {
    36.             // GI hotfix
    37.             for (var i = 0; i < trackConfig.TrackList.Count; i++)
    38.             {
    39.                 var track = trackConfig.TrackList[i];
    40.                 EditorUtility.DisplayProgressBar("GI Hotfix", "reverting " + track.name, ((float)i / trackConfig.TrackList.Count));
    41.                 EditorSceneManager.OpenScene(track.Path);
    42.                 Lightmapping.giWorkflowMode = Lightmapping.GIWorkflowMode.Iterative;
    43.                 EditorSceneManager.SaveOpenScenes();
    44.             }
    45.  
    46.             EditorUtility.ClearProgressBar();
    47.         }
    ]

    edit: I reported it as a bug, hopefully someone can make sense of it
     
    Last edited: May 11, 2017
  12. Jesper-Mortensen

    Jesper-Mortensen

    Unity Technologies

    Joined:
    Mar 15, 2013
    Posts:
    232
    Ideally if it baked before it should also bake with a newer version of Unity. However, maybe the size of the scene was borderline and something pushed it over the edge. There are some useful tips here on how to optimize for faster lighting builds:
    https://unity3d.com/learn/tutorials/topics/graphics/introduction-precomputed-realtime-gi

    If these tips do not help it would be great to get a project from you stripped as bare as possible so we can isolate the issue and fix it.
     
  13. Lost-in-the-Garden

    Lost-in-the-Garden

    Joined:
    Nov 18, 2015
    Posts:
    176
    Maybe it's linked to the out of memory errors we see in the console during the visibility phase. It does not seem to bother the editor though. Maybe some error handling went wrong somewhere and instead of reporting the error it simply hangs.

    We never saw any memory related issues in 5.5, no crashes but also no high resource consumption.

    Since we are getting close to cert, we had to cancel our 5.6 update because we simply don't have the time now to spend on fixing all the problems. We'll try to try the update again though, and then hopefully also have the time to come up with a repro project.

    btw: why is there a seemingly arbitrary limit on the real time resolution? Our settings are quite low, but we would like to go even lower for our environments, but we can't.
     
  14. MrDude

    MrDude

    Joined:
    Sep 21, 2006
    Posts:
    2,569
    Thanks for the link. That proved super useful. I imported the free rocks asset from Manufactura 4K and the scene looked beautiful so instead of creating one from scratch I used multi-scene editing and just dragged it all over to my main scene. From there I deleted about 70% of everything since I have a static scene and most of it won't be seen.

    Voila... new lightmaps need to be generated and I am stuck on 7/11 for days. No clue why. As I was trying to figure this out it seems my terrain shifted ... odd. Even odder still was that my grass still appeared where it should buut my trees are floating about 10m off the terrain surface. Shift clicking on the terrain clears them and clicking on the terrain places them on the terrain again but that was just weird. I am assuming that something that I changed changed a highmap or something...

    Anyway, following along with that tutorial I got to the part where they say to look at the scene using the UV mode and when I did so I noticed that this terrain that was moved to way under my other scene objects, when I moved it back up to where it should have been, now suddenly left me with loads of rocks BELOW the terrain. So I started clearing those objects away and ended up making my 236 jobs that it was stuck on for 2 days into 64 jobs that finished in about 2 minutes.

    Since I had removed all reflection probes and lightprobes from the scene (like I said, static scene during gameplay) this leads me to believe one of two things to be the cause of this hanging lightmap compilation:
    1. Too many objects overlapping and going inside other objects is frowned upon. Especially rocks rotated into such an angle where the hole in the base in visible. Mesh inside another mesh + hole in the mesh receiving light.... Hmmm.... Bounced lighting off of THAT... Hmmm

    2. Calculating bounced lighting on static objects UNDER the static marked terrain is frowned upon.

    Not sure if removing specific rocks or if removing everything under the terrain fixed my problem but whichever it was it worked. Only noticed the issue due to the UV mode. So I say again, thanks for that most useful link. Definitely bookmarked it for future use... even though it kept on talking about realtime light resolution and I could not find that for the life of me. All I found was bounced light resolution... Is that just a name change in a new version of Unity or is that something else completely?
     
  15. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,633
    Still getting this in unity 2017.1.0f3, but now it dies building scene and doing 2/11 layout systems. :-(
     
  16. Lost-in-the-Garden

    Lost-in-the-Garden

    Joined:
    Nov 18, 2015
    Posts:
    176
    just tried the upgrade to 2017.1.0p2 and the issue pops up again. Tried it on two different computers, one even with 32gig of ram. It just hangs during the build.
     
  17. UnityLighting

    UnityLighting

    Joined:
    Mar 31, 2015
    Posts:
    3,874
    Always try Very-Low preset for first time baking .
    For example for my simple room :
    ~2 hours with high settings
    ~ 10 min with very low settings

    So if your level is too large, so you may needs to wait a week to complete baking !!!
    I believes unity Enlighten was OK from start but its settings was not optimized from the beginning without any tutorial
     
  18. no00ob

    no00ob

    Joined:
    Dec 13, 2017
    Posts:
    65
    I fixed mine in unity 2018 by clearing the Lightning Gl, I went into edit > preferences > Lightning Gl cache and clicked clear, after that it baked properly.
     
    SkyLimitStudio and djwhiting like this.
  19. djwhiting

    djwhiting

    Joined:
    Sep 12, 2014
    Posts:
    1
    Perfect, solved my issues!
     
  20. 40detectives

    40detectives

    Joined:
    Apr 9, 2016
    Posts:
    74
    The thing is, the user interface of the Lightning settings have change a lot in 2017 and 2018.
    Having a lot of troubles guessing which old settings are related to every new option.

    It would be too much to ask to update that tutorial?

    EDIT: It seems Indirect Resolution setting (2018.1.0f2) is the one that should be between 0.1-0.5 or 0.5-1 for outdoors and terrains respectively.
     
    Last edited: May 16, 2018
    NEVER-SETTLE and KarlKarl2000 like this.
  21. KarlKarl2000

    KarlKarl2000

    Joined:
    Jan 25, 2016
    Posts:
    606
    40detectives likes this.
  22. amosvorn

    amosvorn

    Joined:
    Oct 31, 2018
    Posts:
    5
    Its easy on all the scenes, go to window-rendering-lighting, uncheck Auto Generate at then click the drop down, click clear baked data then click generate lighting, repeat for each scene, :) (you might not need to clear baked data but i did)
     
    sam031026 likes this.