Occlusion culling CPU spike in profiler

Hi, it seems that occlusion culling in Unity 5 creates unusually large spikes on the CPU in the profiler. I've noticed this problem in multiple scenes, in multiple projects, and have reproduced it in a simple scene with a few cubes in it.

There is only one camera, one directional light and 9 cubes using the standard specular shader. The cubes are marked as occluder and occludee static, and the occlusion culling settings are default.

Here is a screenshot of the scene :



I saw a post with a similar problem, which was solved by removing occlusion culling on a second camera that should not have used it. However, in this case I only have one camera, and it needs occlusion culling.

 

I'm also encountering this spike caused by WaitingForJob (though mine is under Culling/SCeneCulling/CullAllVisibleLights/WaitingForJob).

I need the occlusion culling as I am creating a large detailed environment with many rooms, so clearing the baked occlusion is not an option. I'm going to try being more selective about what I flag as Occluder Static and see if that helps, but I have my doubts. I'll post my results here.

 

Quick update: no, reducing the number of occluders made no difference.

 

Having the same problem myself.

Checking the profiler in an attempt to optimise my scene, and noticed that whenever the camera turns, there is an enormous spike in CPU usage coming from Camera.Render/Culling/SceneCulling/WaitingForJob. Whatever it is, it seems to take up 87% of the CPU for a brief moment, dropping the framerate from a smooth 70 to 10 or lower.

 

I have the same issue, even if I remove Occluder static, and ocludee static from all objects, turn off occlusion culling on the camera and clear the oc data, I still get a spike every minute or so.

 

Hmm. I had initially assumed it was something to do with my DetailGroup scripts (hiding and unhiding groups of objects based on distance to reduce distance calculations) but evidently not if others are having the same problem.

 

I submitted a bug report. I will let you know what they say.

 

We're seeing the same issue popping up in random spikes, interested in seeing if this is a bug or just something that we can optimize somehow.

 

Still no word on their part? I have been optimising my project across the board and have it running at 120fps, but these occluder spikes are still causing dropped frames in there.

 

No reply as of yet.

 

Taken a closer look at the profiler; it seems that the WaitingForJob process that is causing these spikes is actually running on every frame, albeit at a much lower CPU usage. It runs constantly at about 0.5ms, spiking to 20ms at seemingly random intervals. This background process takes up something like 11% of the CPU usage, in an 80fps scene, with the spikes taking up over 80%.

 

Had a look around similar threads; according to this guy: http://forum.unity3d.com/threads/un...lling-stuttering-waitingforjob-spikes.298957/ The bug is even happening in the Unity Viking Village that came as an example scene for Unity 5.

 

I also submitted a bug report, and attached a basic scene with nothing but cubes that shows the glitch. Hopefully that should do it.

 

We are seeing stalls in Camera.Render->Culling->SceneCulling->WaitingForJob - our game is stalling once every couple of seconds (the stalls are for 10-16 seconds). We re running Unity 5.0.1

 

I think we need someone with a pro account to submit a ticket.

 

We made this problem go away by a combination of setting the Bounce Intensity on all our dynamic lights to zero (I think this stops them contributing to global illumination), and not continually enabling and disabling lights (just turning there Intensity to 0 rather than fully disabling them seemed to improve performance).

 

I found that unmark light disabled/enable during runtime as static dit the trick for me.

 

I have the same problem in 5.0.2f1 and 5.1.1f1. Although my game is 2D without any light sources so I guess I have to fiddle around and see if there's another solution. I'll make sure to come back here if I find anything.

 

Oh! I thought it was because I switch off static gameobjects that was baked in occlusion data. Is it still a bug?

 

I am still seeing these spikes as of 5.2.1p1.

Please note that 'WaitingForJob' is a very generic token and does not really identify a specific problem. We need to dig deeper to look at what job (from Unity's multithreaded job system) is taking longer than we think it should. To do this, when you are looking at a CPU frame in Unity's profiler, click the drop down in the upper left corner of the bottom pane and select 'Timeline.' In the timeline view, you can see what is happening on each thread, in actual chronological order from left to right. You can zoom in/out with the mouse scroll wheel to get a better look at the names and timings of specific markers. If you zoom in so far that you can no longer see the entire frame, there is a scroll bar at the bottom of the pane that lets you pan left and right.

Anyway, in this comment I attached an image of a spike frame in a profile I just checked. Have a look!

In this frame there are two problem areas that are causing large WaitingForJob stalls, which I've circled in red. The first one is caused by long CullSceneDynamicObjects and CullDynamicObjectsWithUmbra markers. The second one is caused by long Shadows.CullShadowCastersLocal and Shadows.CullShadowCastersWithoutUmbra markers. Note that all of the smaller marker boxes around those markers are the exact same types of jobs, they just normally take a lot less time to complete. On a non-spikey frame, they are all around the same short duration. Also note that there are different spike levels on different frames. The worst frames are as illustrated in this image, where a job in the dynamic object occlusion culling system has spiked and also a job in the shadow caster culling system has spiked. In other frames, only one of those two subsystems have a job that spikes, resulting in a spike that is not as severe as when both have spiked.

I don't have any solutions to offer. I haven't really started experimenting with content to see if there is anything obvious that could be causing this.

Also, I haven't seen any patch notes that mentioned this, but @karl.jones mentioned in another thread that Umbra was updated in 5.2.1p2. It didn't sound like the update involved anything related to performance though, so it probably did not affect this. Karl, any chance you could poke around into this one, too?

EDIT: I should probably add, in case it matters: this is using legacy deferred rendering for all cameras.

 

Ok, actually, the more I look at other profiles of spikey frames, the more I think maybe the problem isn't specific to any system. I see occasional weird frames with a job that take 20-30 times longer than typical in pretty much all the systems that have jobs on worker threads. For example, I also see (though much more rarely), super large spikes in PhysX jobs:

Code (CSharp):

1. PhysX.Sc::Scene::ccBroadPhase,
2. PhysX.Sc::Scene::postBroadPhase,
3. PhysX.BPWorkTask,
4. PhysX.Sc::Scene::islandGen
5. (bunch of other Culling related tasks)
6. etc

Maybe there is just a more general problem with the job system and/or an issue with memory access or management, and the culling systems exhibit problems more frequently simply because they take advantage of the job system more heavily than any other systems do?

 

We experienced more or less the same spikes as you describe LeonH (WaitingForjob taking 20-30 times longer at random points), and after some investigation it turned out to be a thread preemption problem in our case.

We had a higher priority thread running on the same cores as the worker threads and this thread could take up to 4-5ms to complete its task before relinquishing control back to the thread scheduler. If a job had begun on one of the shared cores, and had been preempted by our higher priority thread, it would intermittently stall the main thread with a long running WaitingForJob.

I don't know if this is the same in your case, but I thought I'd share my findings

 

Hey thanks! That's really good info to keep in mind. In our case, we aren't intentionally doing anything outside the main thread...I guess maybe it's possible something in our game code is causing Unity itself to kick off some work in another thread, but I don't think that is happening.

Actually, my results are from a console platform and I am a relative n00b to anything outside of standalone...Thinking more about what you said, perhaps something in the way the platform's dev/testing environment is set up is causing this and it's not an actual problem in a release/retail environment. I'll have to investigate that.

 

Our problem occurred on console too. The SDKs for most consoles usually ship with profiling tools which make it pretty straightforward to profile/debug this particular behaviour - just sample over a suitable time interval and then look at the CPU timeline to identify thread work.

 

I got this as well, making my game unplayable.

 

Any updates on this?

 

This still happens on Unity 5.3.1p1. Any words from Unity?

 

Any update on this? I dont even use occlusion culling or have any objects marked static, and im getting hits from 30-100ms on 5.3

 

Frustum and Occlusion Culling is 100% broken in Unity 5.2 and 5.3... It's incredible when Culling takes more than a Drawing object.

I have the same problems and i dont know how i can public game with low FPS.

 

I noticed my issue was coming from LOD Objects not culling. Might be everyone elses issues as well. As soon as I turned off Occluder Static it went back to normal.

 

You mean that Occluder Static objects should not use LOD?

 

Idk, i just know it fixed it for me. They still occluded fine, etc. But it certainly got a normal frame rate back.

 

I am having this problem on IOS. Even in a completely empty 2D scene, without any game code running, there are spikes in WaitingForjob regularly, taking 4-5ms on an iPad mini2. By turning off occlusion culling, those spikes go down to 1ms, but they are still present.

 

Any word on this?

 

Getting the same massive spikes too, and I am not even using culling... 5.3.1p1

 

I have exactly the same problems as Jaysta.
We are not using Culling on our cameras, it still happens though. Weirdly enough, I am able to play the game for about 5 minutes before it starts spiking. I have no idea why it starts spiking or how to fix it.


You can see here that it says that it is the Water Camera, but removing that and having no water, just transfers the culling spikes to the Scene Culling...

It is also resolution independent, as changing my screen resolution from 4k to HD doesn't change anything.

I am running this on my Windows7 PC, and all other PCs we tried with W7 have the same problem.
My laptop is running it in Windows10 and I do not have any spikes there. We also have another computer running it on Windows10 and we also do not see any Spikes.

But I can not confirm if it is a problem with Windows7... I need to test it on this machine first. I am probably going to have to purchase another Windows10 License and install it.

 

Sadly enough we're also seeing the same Culling spikes in our project. We'd be very glad if someone can take a look into this since this is causing major setbacks for us.

We're currently making use of 5.3.2p1.

 

Just as an addition to the weirdness, I am seeing a huge spike when turning the view, and it appears related to ParticleSystemPrewarm. None of my particle systems are set to "prewarm".

 

Add me to the list seeing this -- in my case I also saw the ParticleSystem.Prewarm spike.

 

Yes, we sometimes also get the ParticleSystem.Prewarm spike, aswell as Physics and Cloth physics spikes.

BUT! When switching to Windows10 all of these problems go away. Any idea why? I switched to Windows10 on my machine and all of the spikes are gone. Just like on all other Windows10 pcs, there is no spikes.

 

It would be great if someone can shed some light on this issue.

I can confirm what Olafson said about Windows 10 and Culling spikes.

 

I am not confirm. I have Windows 10 and culling spikes.

 

Argh. I wish we could actually get a lead on why this happens...

 

@Olafson

I can`t know about your project. But in our project we have 10-30 thousands static and dynamic objects (5-10 thousands is SpeedTree). That frustum and occlusion culling is big impact on performance with stuttering and spikes.

 

@Morfeuskiev

We do not have that many objects, but in worst case, we do have an ocean renderer, two terrains (one for the seabed and one for the coast) A couple of Speedtrees, a bunch of rocks and small objects as well as houses. But everything is very optimized and we mesh combined as many objects as we can.

In best case, all we have is a few ships and an ocean renderer, and yet we still get those spikes.

 

Bump! Occlusion culling takes more time than everything else on 5.3... Has anyone submitted a bug report?

 

It is not only Occlusion culling for us but even Frustum culling causes huge spikes.

 

Where is Unity support? They does not reading forum?

 

We've upgraded our project to the latest patch release 5.3.2p3 but unfortunately this didn't make too much of a difference.

We're still seeing massive Culling and Physics.Processing spikes when profiling our build.

 

There are two (maybe more) bug reports on this:
https://issuetracker.unity3d.com/is...rmation-is-garbage-collected-when-not-desired
https://issuetracker.unity3d.com/issues/culling-waitingforjob-performance-spikes-in-the-profiler

whereas the 2nd one is set to fixed in 5.4. But it also says that it would not happen in an empty scene, which is not true.