Huge Camera.Render spike on iOS device

I asked this over at Unity Answers, but no bites, so I figured I'd try it here. I apologize if this is bad form!

I have an iOS game that will occasionally freeze, but only on the device, never in the editor. It's not consistently reproducible, but almost always happens eventually - sometimes soon after launch. When I profile one of these crashes I see this:



There is a crazy spike in Camera.Render, but it is all in Self, so what is causing it? There are no accompanying error messages. This usually causes a crash for memory reasons, but not always - sometimes the game recovers.

What can I do to diagnose this problem? What does it mean when camera.render is at 98% but none of the sub processes are more than 0.2%?

Thanks!

 

Do you have a screenshot of what you are rendering?

 

Here it is. The explosions are particle effects, using the mobile/particles/additive shader. The reflection is created by a second camera rendering to a render texture. The monsters have sprite animated tentacles.

The weird thing is that the game can run with many explosions happening all the time for literally hours with no problems. Days go by of fairly heavy testing with no crash. Then it just starts happening. Sometimes it starts happening straight away with a fresh install of the same build that was just running fine.

Using Unity 5.3.3, but this crash has been happening since 5.2 at least. Unfortunately I don't know exactly what version as I attributed it to too many draw calls or generally inefficient texture management and put a lot of effort into fixing that, with generally beneficial effects but no impact whatsoever on this particular crash.

Thanks for taking a look! Really stymied here.

 

hmm let's see...I believe there were some issues with particles and batching in some of the unity5 builds.

Somet things you can try to start with:

Read around on the forums if the particle system bug affects your unity version

As you also point out that it happens on device only...if you are running it on an iOS device either iPhone5S/ipad mini with retina or newer..then you could try not using the "metal" graphics API...to see if there is anything exciting going on with that..pick OpenGL ES 2/3.

Also try to build a non development build... and make sure to change the Scheme in Xcode to Release and untick debug.

 

I would suspect that performance spike is related to too much overdraw in scene, it could be particles, but also it could be just too many layers of transparent geometry. What Xcode performance gauges are showing for CPU and GPU load?

Regarding crashes.. we need to see some of stacktraces, where the crash is happening, to give reasonable advice where to look further.

 

Thanks sfjohannson! Trying a build with your suggestions now. As the crash can be infrequent it could take a while to know if this has made a difference. I will report back either way!

Thanks, Mantas! That was my original thought, but there are not very many layers and not a lot of transparency - I weeded a lot of it out. Also the game can be a lot busier than shown in this screenshot, with many more sprites and particle effects, without even slowing down. It is as likely to freeze on an early level with not a lot happening as on a later, crazier level.

The game does not always crash when this happens - the frame rate slows to a point where it is essentially frozen, but occasionally it does calm down and recover after a few minutes. When it does crash the stack trace usually shows something like:

Exception Type: EXC_RESOURCE
Exception Subtype: WAKEUPS
Exception Message: (Limit 150/sec) Observed 206/sec over 300 secs
Triggered by Thread:14 (this varies)

Which based on my research seems to indicate a general memory overload problem. I don't have the figures for CPU and GPU load currently - next time it crashes I'll get them!

 

Using OpenGL ES 2/3 does make a difference, but it doesn't fix the problem. The slowdown still occurs, but it is less drastic - gets to about 1fps rather than a complete freeze, and doesn't fully crash. Still unplayable, unfortunately, when it happens.

Does that suggest any possible further action? Is this looking like a Unity bug, possibly something to do with particles, or is it something I have done badly? I would prefer the latter!

 

If it still drops with OpenGL then it's probably not related to the implementation of the metal API...so still something Fishy with the actual stuff..

What stats do you get in editor...when u click on the button in game view..?

One simple test you can do is to make a build without the particles...and see what happens...
Also one without the reflection..

 

This has never happened in the editor. It's not triggered by a specific button click - we have never been able to isolate a specific circumstance that causes it after weeks of trying, so we can't replicate it in the editor in any meaningful way.

I'll try different builds without particles and the reflection. Does the fact that the number of particles on the screen doesn't seem to correlate to the crash suggest that it isn't the particles, or could it be anyway?

One thing to note is that didn't happen for a long time - the game has had many particles and the reflection since the beginning, but these crashes started appearing fairly late in the process - a few months ago (we have been working on this project part time). This led me to believe it was related to adding more art, but that does not seem to have been the case.

I made the rookie mistake of updating Unity whenever a new version came out, which led me to updating to patch builds when the new versions had bugs that affected us etc., so I think perhaps it could be a Unity problem? Perhaps not with Metal implementation, but with something else? Currently we are running 5.3.3f1.

Thank you for your help!

 

Even if the crash don't happen in editor..we can get some sanity check numbers....the scene looks simple but to just to make sure...

There isn't that much on screen, that's why first guess was that it was related to the particle batching and unity5 as you said that it did not use to happen( I have not experience with the particle bug myself as I use unity 4 for iOS)

But I'd say try without the particles...they look a little "dense"

You can also use Xcode to capture the frame:

GPU Frame Capture: Debugging a Frame on an iOS Device

 

Trying the build with no particles over the course of the day. Here's a capture of the profiler from the editor of a busier frame. I turned off the Vsync and Others items as they were causing huge spikes just waiting for the next frame (target frame rate set to 60fps).



And here's the screenshot.



Apologies for the low resolution - small monitor.

 

The game crashes the same way in a build with reflections and no particle effects, and with a build with particle effects and no reflections, so it seems to be unrelated to either of those systems.

The weirdest thing to me is the way the profiler behaves - the way Camera.Render spikes, but without any of the underlying processes spiking at all. What could cause this to happen? Is there a hidden process?

 

Hmm well...I think you at least tried eliminating the possible trouble makers...unfortunately without success. Maybe you have to dig into the wonders of Xcode debugging more..

This this might give some clue:
http://stackoverflow.com/questions/...th-exc-resource-wakeups-exception-on-ios-8-gm

Somebody else is probably better suited to give advice on how to dig deeper...

 

Yeah, I spent a while trying to find a solution to that Wakeups Exception thing, and the closest I could find was advice to reduce memory usage, which I have done to no success. The weird Camera.Render spike in the profiler makes me think it is something within Unity.

Thanks for your help!

 

The last post in that thread points to something with OpenGL....which sounds like a possible culprit here..so maybe dig through that and see where it takes you!

Some exciting shader...image effect...that sneaked in...that you don't see but it's there..or as you say something in unity that you managed to expose as the planets aligned.

http://stackoverflow.com/questions/19506797/ios-7-and-opengl-issues-crashes

Best of luck..may the force be with you!

 

Here is a frame from the Xcode profiler just before a freeze:

iPhone Unity internal profiler stats
cpu-player> min: 3.9 max: 9.8 avg: 5.9
cpu-ogles-drv> min: 0.0 max: 0.0 avg: 0.0
cpu-present> min: 0.0 max: 0.1 avg: 0.0
frametime> min: 12.6 max: 20.3 avg: 16.7
batches> min: 36 max: 40 avg: 38
draw calls> min: 36 max: 40 avg: 38
tris> min: 844 max: 1182 avg: 1081
verts> min: 1500 max: 1852 avg: 1693
dynamic batching> batched draw calls: 0 batches: 0 tris: 0 verts: 0
static batching> batched draw calls: 0 batches: 0 tris: 0 verts: 0
player-detail> physx: 0.7 animation: 0.3 culling 0.0 skinning: 0.0 batching: 0.0 render: 2.8 fixed-update-count: 0 .. 1
scripting-scripts> update: 0.7 fixedUpdate: 0.2 coroutines: 0.1
scripting-memory> used heap: 2449408 allocated heap: 2768896 max number of collections: 0 collection total duration: 0.0


And here the first frame of the freeze, coinciding with the spike in Camera.Render:

iPhone Unity internal profiler stats
cpu-player> min: 4.2 max: 1024.3 avg: 827.8
cpu-ogles-drv> min: 0.0 max: 0.0 avg: 0.0
cpu-present> min: 0.0 max: 0.1 avg: 0.0
frametime> min: 14.9 max: 1025.2 avg: 830.9
batches> min: 20 max: 39 avg: 28
draw calls> min: 20 max: 39 avg: 28
tris> min: 474 max: 1063 avg: 715
verts> min: 651 max: 1591 avg: 1048
dynamic batching> batched draw calls: 0 batches: 0 tris: 0 verts: 0
static batching> batched draw calls: 0 batches: 0 tris: 0 verts: 0
player-detail> physx: 4.5 animation: 0.6 culling 0.0 skinning: 0.0 batching: 0.0 render: 785.7 fixed-update-count: 0 .. 17
scripting-scripts> update: 0.7 fixedUpdate: 1.2 coroutines: 0.4
scripting-memory> used heap: 2449408 allocated heap: 2768896 max number of collections: 0 collection total duration: 0.0

CPU usage went from about 60% before the freeze to 6% during it. Memory was around 250MB.

Does this suggest anything to anybody? Thanks!

 

Try to do also do a GPU frame capture...with that you can see exactly what is being drawn...object by object..shader by shader... which you also ca save and go back and look at it later... btw which iOS device is it?

Here is an app that you build that ups the memory usage until it crashes, to see if it reaches the same threshold as your app..before it tosses in the towel.

https://github.com/Split82/iOSMemoryBudgetTest

 

It's a 6s Plus, so plenty powerful. Unfortunately GPU frame capture doesn't work once the freeze has happened.

 

Exactly the same issue! I have 2-3 particle systems in frame, one of them emits alpha blended & Sprite particles. The game just randomly hangs and nothing happens. That's really bad. The release is soon and it would be f***in awesome to be stuck with a bug in Unity. Of course the problem is in Unity since the performance drops very . However the music plays and sometimes game jumps few frames ahead and freezes again. Have you found a solution?

https://github.com/Split82/iOSMemoryBudgetTest results
Crash at 1394 Mb, warning at 1115 Mb.

My Unity version is 5.2.3f1. Test device: iPhone 6s.

 

Hey! Thank you for prompting me to share with this thread - I had completely forgotten about it (I am aj-highline - different job, different account, apologies for any confusion).

Yes, we did eventually track it down, but it was very specific and I don't know if it will be helpful for you. Fingers crossed!

It turned out to be nothing to do with any memory issue at all - it was caused by incorrectly setting up HeyZap, which we are using for ad mediation. We noticed that the crash always occurred a set amount of time after launching the app - I think it was 2 minutes and 15 seconds. Very specific, very repeatable, but not actually very noticeable - until suddenly it becomes completely obvious. Once we noticed this we deduced that it was some process that launched when the app started, and a process of elimination led us to HeyZap.

If you are using HeyZap, go to the install wizard page, select all the services you are using and then follow the instructions carefully each time you build - we were skipping a necessary step due to not selecting all the services when refreshing the page. If you are not, I'm afraid you are having a different problem. But do see if it happens after a fixed amount of time. I don't know why a failure in HeyZap caused this kind of crash, but maybe if there is any kind of web service that you try to connect to at launch it could be causing this.

Good luck!

 

Indeed, it seems I was accidentally stacking Heyzap banners every restart creating a new instance and was ending up with approx 25-30 banners in the bottom of my screen. Oh what a shame. Thank you so much al-gco for your hint!

But how is this possible? How 25-30 AdMob banners were able to crash the game? And why Unity profiler was showing such a huge performance spike on Camera.Render?