Zombieville USA - performance hitches

So I've released my app, Zombieville USA, on the app store and its doing fairly well so far and getting good reviews, but there's this periodic framerate hitch that I've been trying to track down ever since submission.

The Problem: Basically every few minutes or so, the game will stall briefly (sometimes the stall will last for a fraction of a second, sometimes for a full second or two). Both the visuals and the sound will freeze during this time (especially noticable if there's a sound effect playing at the same time - it will hang in the middle of playback in a most noticeable way). This stall is then followed by a 5-6 second period of decreased performance overall - the framerate is lowered, and the responsiveness of the touchscreen goes to hell, almost like its recovering from something. In a fast paced action game like mine, this is really annoying, since during this period of decreased performance, the fire button becomes unresponsive. Freshly restarting my test device seems to reduce the intensity/frequency of these stalls and their following period of poor performance, but they always occur eventually.

But wait, there's more! I first discovered and reported this issue a while back, here's a link to that thread.

http://forum.unity3d.com/viewtopic.php?p=120927&highlight=#120927

For some strange reason, my game was experiencing this same stall every 15 seconds or so after I added a 'menu' level to my project, rather than every few minutes. I ultimately removed the menu level, and simply incorporated its contents into my main game level, which solved the extremely high frequency of the stalls.

So what's up with my game? These stalls occur regardless of what's going on on screen - it can be in the thick of battle with a dozen zombies, or just walking down the street with only one enemy on screen. I suspected garbage collection was the source of these issues, so I tried putting a request to run GC at the start of each stage. This seems to have had little or no effect. I also completely re-wrote my zombie spawning system to use a pool of recycled zombies that are always in memory, but hidden off screen until they're needed, rather than constantly instantiating and destroying them. This helped performance in general, but did not remove the stalls. Overall the game runs pretty well, but this issue is driving me crazy, and seems completely random and out of my control.

I'm going to try sending in a bug report with my project so the Unity folks can pick it apart.

 

I don't really have a solution for you, but I do want to say that our game Sheepstacker suffers from the same thing. Every few minutes, we get some pretty bad hitches. The sheep are all pooled (just like your zombies), so there's not really any real-time instantiation going on.

I believe in our instance, it's definitely garbage collection. On level load, I'm requesting that garbage collection be performed, but I don't really think it's helping. But looking at the printout from the profiler, it looks like the heap is getting freed up right when the framerate hitches occur.

 

I had this happen with Downhill Bowling sometimes as well. Maybe it's something to do with Unity? Perhaps a bug that still needs to be fixed? Other than that, I have no idea for these weird stalls.
-Blayke

 

this error is caused because of the garbage collector that cleans memory when you are in the middle of the game, I sent a feature request some weeks ago in order to have a function that forces the garbace collector to collect when I want but the only one thing I received as a reply was "could you be more specific" and closed my ticket. (Someone who has knowledge about programming would understand I think).

I recommed not to use ArrayList because this causes a constant 4K increase in memory use and not to instantiate objects every moment.
When your game begins, create 5 or 10 zombies and put them in a fixed array, cyclically use each monster

 

I am already doing this, as described in my original post. Using a pool of enemies in a built-in array vs. instantiating/destroying them had no effect on these stutters. I am using javascript arrays to handle my particle system though, and frequently using .Push() and .RemoveAt() - if that's leading to rapid memory consumption I'll revisit that code and have it use built-in arrays instead.

As for ReJ's questions - the stats readout during these spikes aren't very telling. The cpu-player max time spikes up to a huge amount, but nothing else exhibits any unusual behavior.

I'll do some tests tonight and look for "WARNING -> applicationDidReceiveMemoryWarning()." This all rings of a memory issue, and I've gotten reports of crashing from players using jailbroken phones... and those players were able to resolve the crashes by disabling some of their memory hogging mods/software.

If I am riding the edge of how much memory I have available, what is the best way to reduce it? Would reducing the size of my assets be the major factor, or is it more likely to do with my scripts? A lot of my textures are pretty high-res, I have some 512x512 ones for UI, background art, etc. Are variables that contain references to other gameObjects, textures, etc. costly as well? There are a lot of holes in my understanding of memory usage in Unity, what's expensive vs. what is negligible or 'free'.

 

Thanks for the info ReJ, that helps clarify things. I'm probably strangling the poor iPhone with my hi-res sprites, since I've been very reluctant to use PVRTC compression. With textures on 3d models it isn't as noticeable, but when you're using a crisp, hand drawn art style like mine it really destroys the visuals with ugly artifacts. I'll take another pass at my textures though and make sure none of them are larger than they need to be, and see where I can use PVRTC without being highly noticeable.

My sound effects are minimal and very compressed, my music is an mp3 which could probably be compressed further.

 

Remember that the iPhone resolution is only so large, and you never get any closer to your artwork than the plane the action resides on...

...since there are only so many pixels, there's isn't a need for FULL SCREEN texture to ever be above 480x320, let along a single element that is merely a portion of the the screen; you can't draw more than the pixels that are already there.

I know this all goes without saying perhaps, but then again you ARE using 512's.

 

Just because a texture is 512x512 doesn't mean you're seeing it all at once, or even on one contiguous object.

For instance, I have a 'store' in my game that uses 16 different icons, only 5 of which are displayed at any given time, but all 15 icons, their price tags, and the building graphic that surrounds them share a single texture.

 

If it's an atlas fine...hell I'd use the entire 1024x then...

...either way I misunderstood what you were using them for.

 

i thought vram didnt count toward your mem footprint?

 

I'm pretty sure the iPhone has no dedicated VRAM, so textures would have to draw from the same limited memory that everything else is.

 

I agree that uncompressed images are likely the issue. I'd copy off a test project and compress all the images, than see if the freeze-ups disappear.

Any time I've tried to use relatively large uncompressed images on the iphone, I've had crashing problems or freeze issues.

I could see the garbage collector firing constantly in an effort to free up memory in those constrained conditions. Even with pooling, we do not know what memory is being allocated behind the scenes by the UnityEngine. In my experiences with java's VM, if memory usage is close to its max allocation, it will churn trying to free what little objects it can.

 

An uncompressed 1024x1024 texture is about 6MB of graphics RAM, which is 25% of what you have at best to use for your graphics.
So thats definitely a no go

 

For those who are curious, my texture sizes turned out to be irrelevant, but it definitely seems to be something memory related. I ran some tests with activity monitor, and my app's memory usage very quickly swells during play, even after all assets are in memory. It even increases while the game is paused and nothing is really happening... eventually the memory usage hits a plateau, and that's when the stall occurs. The stall and the memory growth stopping always coincide, but its not a specific ceiling. I tried compressing the crap out of my textures, and that simply made the stall occur 10mb's earlier. It didn't matter if my app was using 30, 40 or 50mb, the first stall happened like clockwork within a couple minutes of starting a game.

I've submitted my project for dissection, hopefully its just something stupid that I'm doing.

 

Sometimes it's not so clear because the os can cache memory and release it back to the app when it requests it, so it isn't unity or possibly even your app related.
Like for instance in a game I have I use nsfilemanager a lot and I'm always running with @ 3 mb to spare and when I drop below the level to trigger mem cleanup (3 or 2mb I think) I'll mysteriously bounce back up and have more memory. It doesn't mean my app is to blame. It my case the men was released long ago but the mem was never returned to my app till it needed it.

Doesn't mean this applies to you for this issue, but it's good to keep in mind.

 

Dissection by whom, may I ask?

 

Yep! All the same RAM. Textures, program, background processes, OS, sound files - all figting for the same RAM.

At the Apple road show tour they preached an approx max memory footprint of a running app at _max_ 25-30 MB. Anything above that and you into the danger zone for getting memory related problems and eventually forcibly shutdown by the OS watchdogs

 

I just sent it to the Unity guys in a bug report. After thorough testing I'm pretty confident that this is not something I can fix. I hope to be proven wrong though.

As for the 20-30 mb max, that's kind of unreasonable. My game is using 20mb after loading nothing more than the main menu, so by that logic a unity game is almost at the threshold of "too much" with nothing but an empty scene.

As I stated before, compressing my assets down to an almost incomprehensible, low-res mess had no effect on the problem I'm facing.

 

We really need more frequent patches for unity iphone to get these things fixed. On the desktop environment these memory/preformance issues aren't as big an issue but developing on iphone every performance gain/ memory tweak that us developers can get our hands on is crucial! just my 2cents

many thanks to PirateNinjaAlliance for submitting your project to unity.

cheers
Jonas

 

This is just a shot in the dark, but in Xcode what version of the device are you targeting? I always still use 'Device - iPhone OS 2.0', rather than 2.2. Does that make any difference?

 

Yeah, I've just been using 2.0 as well, doesn't seem to make a difference.

 

I am simply quoting the Apple tech guys holding the presentation!!

So if empty scenes run 20 MB, then you got max 10 MB for your game before getting into the danger zone.

So there seems to be a very urgent need for UT to shave memory footprint down on the base engine.

 

Wow, quite a bit discouraging, especially as of a few days ago having purchased Unity+iPhone...

...here I was grumbling over the 24 meg threshold I remember reading about, and now it's potentially only 10.

It's not unworkable but it certainly makes a huge difference.

 

Wait sorry, I reread what I thought I read (Programming Guide) and it states that for OpenGL ES apps, you have 24 megs for textures and meshes...

...so the big question is:

Is this 30 megs* (app) + 24 megs (graphics)?

If so this is back to being good again...




*I derived the 30 megs for the app from this discussion and Thomas' last post.

 

Sorry - 25-30 MB max for everything in memory. As there is no dedicated VRAM, everything is in the same pot so to speak.

I have not really found any solid sources of these values besides what they told us at that iPhone road show conference. But it was the mantra of the day really.

Use instruments, use instruments and use instruments - all to track your memory usage

 

Problem is everything I read is anecdotal, so I find it hard to accept that we can't get hard numbers and some predictability here, on top of the fact that the engine takes up two thirds of the total allowed...

...back to not good again.

 

The iPhone has 128MB of RAM total. On other threads in this forum, there has been consistent feedback that we have 20-24MB of texture memory PLUS 40 or so for animations, sound, etc.

I too am only hearing this anecdotally from this forum, so any confirmation of the correct number is really appreciated. This is certainly a big deal...

 

...and these anecdotes include other iPhone dev forums I've perused via a quick Google...outside of Unity.




Oh, and just to be clear, I don't have unreasonable expectations for graphics, as obviously good graphics != fun game, but I don't like the idea of being THAT close to the limit, so this is about function, not merely an unrealistic ideal...

 

Welcome to the world of iphone development in general
The fact is there is a limited memory on the device and when memory runs low, the OS expects all apps to cooperatively release memory when asked, and if the apps don't comply then they are summarily killed off, starting with background apps, and finally with your app.

iPhone SDK apps are also sandboxed, and the virtual memory system on OS works a bit different than virtual memory on OS X. So it's definitely complicated and if you are looking for a "hard number" that is guaranteed across all user devices - I wouldn't hold my breath.

It sure would be nice if Unity had a smaller footprint, but regardless it's still up to the developer to find the sweet spot of memory usage provides a rich experience, but also works on most devices.

 

Absolutely, and I've done well to learn about "expectation" in life (as in not to have any), but lo and behold, I fell into that pit again because it sounded a bit too good to be true...expectation dashed against the rocks...

 

I've experimented with lots of different levels of texture compression with zombieville to see how low I can get my memory usage, and if its around 40mb or less the app never recieves a memory warning from the OS. The current version on the app store uses as much as 50mb at its peak, and its never crashed on me once (although it does get warnings from the OS). I've gotten a small number of crash reports, but they represent 0.1% of my downloads so far, and they've all been from users with devices with no breathing room for virtual memory, or they were jailbroken and running a bunch of memory hogging background processes.

Take that information for what its worth. I certainly wouldn't panic.

And once again, my app's memory footprint has no effect on the problem described in my original post.

 

50 megs is the breathing room I'd hope to hear and I've read anywhere from 35 - 45 generally before the OS might barf it up...so okay, no panicking.

...also, fwiw, bought your game and it is fun, albeit a tad difficult and the weapon response can vary. Interestingly this is where I experienced the slow down, which seemed to be a mix of audio, input and general activity. This happened roughly a 1:30+ into the game. Needless to say I didn't see Level 2 nor another weapon.

Small suggestion:

I sincerely believe it would be satisfying to be able to move in one direction and shoot in the opposite. Would instantly alleviate some of the difficulty, though you might have to add a WalkB anim to accommodate it.

Then simply split your virtual spacebar Left/Right for shooting Left/Right, disengaging it from facing. That or do movement on the left side <-/-> and shooting */* on the right.

Lastly the zombies can get so close you can't hit them with weapons, and they wittle at your health very quickly. Your only recourse then is to walk through them to get into the open, only to have to turn to face them to attack, which starts the cycle all over. A nice bash move, perhaps even a simple dash by double tapping the movement buttons, could get you out of trouble. This way you won't need the bash to do damage, requiring the player always shoot, but at least he can get to some safety, with perhaps a small time before he can dash again (out of breath?).

Otherwise very nice job bro. The pseudo 2D feels great and the animation has nice weight and overlap.

 

Haha, yeah, that's the dreaded lag rearing its head.

The game is way hard, I've already got optional difficulty settings and some other much-requested features implemented, just trying to get to the bottom of this performance hitch before I push out an update.

 

Crap, I edited my post above after you just posted. Reread my last if you're interested.

 

Ok, so I spent the whole day experimenting with the Xcode instrumentation tools and "Free Memory" application.

From a baseline perspective, my main menu clocks in at 14MB of memory usage.

@PirateNinja - I confirmed your figure: my app bombs out when I cross 46-50MB. I actually have a version that has 34MB used and it works well... The version that's at 46 though chugs hard.

Using the "Free Memory" app shows that I have anywhere from 20-46MB free based on the most recent iPhone reset. So while there is a theoretical max of 46-50, from a customer sat perspective you'd see customers crap out earlier because they have garbage still in memory that may or may not be "your fault".

Bottom-line: Unity apps start at 14MB with 'nothing in them' and can work up to 46MB for the "perfect iPhone". However, it's clear that we need to plan on a lower level of memory usage to account for users that don't have all their memory available. What number you decide to pick is based on your own personal threshold for customers satisfaction. Pick 40 and you'll be hitting the wall constantly. Pick 25 and you're doing well.

Hope this helps some folks.

 

Just to clarify, since many people will be wondering this, is this number WITH build stripping (iPhone Advanced) or without build stripping (iPhone Basic) ?

 

That's without... I can't remember exactly atm, but I think with stripping it's 6 or 8MB.

 

Is that stripping the average or an absolute?

 

Well, it can't be the same amount of stripping in each case if you're using certain components. All it does is not include components which you do not use, afaik.

 

WITHOUT build stripping. I'll strip tomorrow and confirm Antennae's 6-8MB amount in my app.

 

Right, my question came out more naive than I intended...

...I guess what I meant to ask is what the average stripping is and based on what expected usage. I did make the assumption that perhaps there were components that you might never use or used infrequently, thus an absolute reduction by X amount.

 

I'm also doing a simple testing with Unity Iphone Basic,
just one cube rotating every frame, no light, no texture, and rotate with simple script like this:

Code (csharp):

1.  
2. var speed = 10;
3. function Update () {
4.     transform.Rotate(Vector3.right * Time.deltaTime * speed);
5. }
6.  

And this is what i got from the Instrument:

All Allocations
- Net Bytes = 9147120 = 8.72 MB
- #Net = 41160 = 0.039 MB
- Overall Bytes = 13378416 = 12.758 MB
- #Overall = 85016 = 0.081 MB

Does that mean it's using 13 MB for this rotating cube ? I have to really careful to use texture later if it is the case.

Also i notice couple occasion of leaks, which have been discussed before several times in the forum.

btw you can find my experiment is in here:
http://forum.unity3d.com/viewtopic.php?t=19612

 

Hey PirateNinja I just saw your game featured in NEW AND NOTEWORTHY on the iTunes home page! Cool stuff.

 

Yeah, I just noticed this myself! They also put a 'Best of the Undead" section in the games category, which includes Zombieville. Pretty exciting, I'm curious to see what effect this has on sales.

I'll hopefully be able to fix this periodic lag and push out an update ASAP to further capitilize on the exposure, as I have a lot of new content added to the game as well, and this framerate hitch is the last thing keeping me from submitting to apple.

 

awesome work pirate. a slightly off topic question - it seems like the "resume" option comes up at random times at the menu screen. not sure exactly when it comes up an when it doesn't.

 

#52 and climbing in the US - way to go!!

I reckon you should publish your update ASAP (I've never even noticed the performance hiccup), this game definitely deserves top 10.