No More "Cool" Features Please Until Crippling Stuff Is Addressed

So, nearly all of the allocation is done up front through object pooling. The first cut scene plays, after which gameplay begins. Memory consumption hovers around the 1.5GB mark with no indication of memory leaks ( we've eradicated most if not all of those ), the gc kicks in and does it's job, albeit slowly. Players often progress to the next cut scene, and as it tries to load the audio from the assetbundle, that's when the crash hits ( with memory still around 1.5GB ).

Memory is mostly consumed by textures and a million nodes of pathfinding grid graph. Now we're ripping out the Unity terrain and implementing our own with a map editor, we're moving to non-unity navmeshes ( implementation is too partial in Unity to be usable ) which will be much more memory efficient. Most infrequent audio has been pushed out to be streamed on demand ( that's the 2GB ->1.5GB reduction and is mostly character voice acting ). That was painful because a) www doesn't currently get the ogg duration from the header so we need to manually specify it so we know when to clean it all up ( no events on streamed audio finishing ) and b) no batch editing of audio resources.

Animations and audio ( 1 master mix wav file including all audio for the scene ) for the cut scenes is loaded async just before a cut scene is needed, and then cleaned up afterwards.

Also, the map in the tutorial is huge. We're taking all our learning from this first phase forward into the next, and will be able to improve the memory footprint further.

 

I've been waiting to grab your game, i'll probably do it now to give it a look.
A few things: why 27meg audio file? That sounds quite huge no? What format is it? Also a million nodes of pathfinding sounds quite large too.
Also from looking at the steam survey, around 73% of steam users are on some form of 64 bit windows, i think a 64 build for that many people is worth it if it fixes it!

 

Yeah I remember reading the same thing somewhere. Unfortunately we can't recreate the issue on any of the team's machines. These are all happening on customer machines so all we can do is up the logging level which is how we discovered what was causing the out of memory crashes.

 

The important takeaway here as I see it, is that you can't use 1.5 GB of memory on a system with 3GB physical memory on windows and expect no problems. And this isn't Unity's problem to fix.

Out-of-memory problems are hard to nail down because they pop up in all kinds of random ways - but the solution is almost always to make budgets for your memory that put you within safe ranges, and then stick with them. If you don't have fixed budgets, and you're determined to support users with memory less than your min-spec, then you're in for a world of hurt. Your memory footprint is 3X that of the XBox 360s total memory; I think it's safe to say you've got some room for improvement.

Some suggestions:
- 1.5 million pathnodes kind of sounds like you have an open-world game and you're loading high-res pathfinding data for the entire world or something.
Some games use systems of pathing where a lower-density version of pathing is used for far-away areas. I'm not sure what's available in the asset store as far as more sophisticated systems like this.

- What's your texture and audio usage like? What kind of budgets do your artists and sound designers have for these, and how are you enforcing it? Are you using mono for all your audio?

If I were in your shoes, my question right now would be how to cut memory usage in half, to open up the game to more players, make loading faster, and make the game more stable for a variety of users. Maybe this will involve more aggressive audio and texture compression, more intelligent streaming, etc. Or maybe it involves splitting your environment up into smaller chunks that fit into memory more easily (see Borderlands for example.) But it's going to be worth it, and probably a much more productive use of your time than waiting for UT to improve their memory allocation in hopes of shaving off that top few MB.

 

@aaronflippo I think I've addressed many of these points in the post at the top of the page.

 

Now before you pick up that cleaver and start chopping memory out of your game in a fit of red rage, I'd recommend turning off a firewall or two if you haven't already done so. There may be a firewall out there that doesn't like unity.

 

Gone through all the typical 0xc0000005 causes and ruled them out with several affected players :-(

Providing a 64-bit version is something I'm going to explore with Valve.

 

Something I'll checkout tomorrow - thanks for the recommendation superpig.

 

Physics.RaycastAll()
Physics.OverlapSphere()
Physics.SphereCastAll()
Physics.CapsuleCastAll()
Rigidbody.SweepTestAll()

GameObject.GetComponents()
Component.GetComponents()
Transform.parent

Camera.allCameras

Light.GetLights()


Garbage generated from Unity internally:

"SendMouseEvents" within DoSendMouseEvents()

 

Post concerning practical memory limits from Aras, Unity's graphics programmer.
http://forum.unity3d.com/threads/162848-Maximum-amount-of-RAM-on-OS-X-and-Windows

Note he says 1.5GB - 2GB on a 4GB machine which is what we've worked to ( ignore the log from the player with 3GB RAM ).

 

This is actually the same for every 32 bit process by default pretty much (but if they really can't port out of 32 bit it would be really nice if they worked around this internally at least for their own allocations, it's probably far from trivial however).

 

Trying to get a 64-bit build of Folk Tale into Steam at the moment. Slight problem is the packaged mono folder is named the same both for 32-bit and 64-bit versions. Would help if they were named differently. I'll also point out that Unity's 10.4 Mac build isn't compatible with Steam's DRM module.

Message to UT: all these "AAA" titles that received millions in Kickstarter funding and that chose Unity are going to come knocking at your door with these issues in the not too distant future. You have an opportunity right now to work with Steam Early Access titles to improve your support for the largest global digital distributor by market share ( Steam ) and be prepared.

UPDATE: Just watched Natosha's video on publishing to Linux and in the last slide she mentioned they'd like to improve support for Steam. Whether this is just Natosha, or UT as a whole, I don't know. Natosha I've added you as a friend in Steam to ask you both about Linux ( we have Folk Tale running on Mint ) and about Steam integration.

 

I grabbed a copy of Folk Tale off of Steam a week or so ago but only got around to trying it yesterday. I liked it a lot, and if there were ever a collectors edition with miniatures of those nasty yet adorable little goblins then I would totally buy it.

It's a game that is generating plenty of buzz and pushing the envelope of what Unity can do. So it's a bit disappointing to see UT not reaching out to the developer and addressing their concerns. I mean this could be one of those games that, when a person asks "can Unity do this or that," you just point at Folk Tale and answer "'nuff said."

 

Not necessarily if they bought Unity with Source.

But on a serious note. UT should really take action and adress all those issues. Folk Tale is one of these games, that are a great showcase for Unity.

 

I think it's worth unity having a bit more time for Games Foundry, so I hope that happens, as pushing unity to the limit can only improve it/expose weaknesses.

 

Our doors are always open UT. I'm very happy to provide feedback if it helps improve the engine. That's all I've ever tried to do here in the forums.

 

If the new Oddworld title experienced all the problem from this topic, maybe UT will do something for all the previous problems.
Anyway, I will continue to follow this topic with the hope to see a positive evolution

 

UT, can you provide us all with an update as to what is happening about this please?

 

Methinks this thread may end up buried like the last one ir the one about asset store reimbursements

 

I hope you're asking them directly as well as posting on the forum. This is nice and visible, but it's not a very effective way to get a quick / direct response. The people who can answer the question simply aren't sitting on forums waiting for them to be asked.

 

David and Joachim are both aware of this, and I've been exchanging a few emails with Joachim who kindly invited me into the beta program.

 

9 months have now passed since the community provided the information requested by UT on which methods in the API were causing unnecessary allocations. A little transparency would be very much appreciated guys.

 

+1

 

+100

 

+10^4

 

+ 9 | | | 9

(imagine the | are up arrows)

 

Did you make an entry in feedback? I prefer it to vote there instead of writing +1 in here, which feels rather pointless.

 

+1

 

The Unity feedback feeback preventing page? I will never understand why i am just allowed to have ten good ideas ...

 

I would call this spam prevention

 

15 Million people in the US visit Wikipedia each month. Of those, 35,000 are active contributors, which is ~1/500 ratio. If the Unity feedback system is like the rest of the world, then most of the feedback comes from a small number of users. Which means, they should add +2 for every month the user has been around. And another +2 for every idea that more than 10 people voted on.

If Wikipedia had similar restrictions, 90% of the content wouldn't exist.

Gigi.

 

The content of Wikipedia is created by users. That is not the case in Unity. We use Unity and can tell them what we would like to have. You can't compare those two at all.

Nevertheless, I like the idea that active users get 2 votes per month or something in that direction. Not sure how hard it would be for Unity to implement that or if it would see other requests in the top ten. There is clearly a chance that it could become more valuable!

 

I have my ten votes placed since two years. And no more vote left for another idea. The ten votes that i have made are still valid to me. So unvote them isn`t a option for me neither. So no, you cannot really give Unity feedback, unfortunately. The Unity Feedback page is more a feedback prevention page to me, which has more an alibi function.

 

It doesn't mean you need more votes, it means the system is pretty useless because things don't get implemented (else most people would get a nice chunk of their votes back).

How many things from feedback (not even % based, how many actual things total) have made it into unity?

 

Has anyone visited UT Feedback lately?

Filter by the different statuses and you'll see that UT hasn't set almost any 'new' feedback to started / completed / planned since ~2011 / 2012.

In fact, there is 1 feedback from 2013 that has been labelled from UT as completed (none for started / under-review / planned), 3 from 2012 as completed, majority from 2011 and 2010 just sitting there.

Although I cannot verify if UT is mostly interested in old feedback (with most votes), between today and one month ago (8th July); ~110 new entries were posted into the system.

The point being, with roughly ~100 new entries per month, ~3000 in the system already, there are only 6 entries in the last 2 (!) years that had their status changed to 'any' other status than active.

We have a saying for this; 'the good idea club'. It just feels like it's a broken way of suggesting ideas and it's not really linked to UT development plans at all (or not properly maintained by UT).

 

We call it the backlog pile. A way to help our inner-perfectionist accept that some ideas won't ever get implemented.

Gigi

 

But he's not comparing Unity to Wiki, he's comparing the feedback site to Wiki, and not only is the content on the feedback site user generated but that's the whole point.

I think the idea with the 10 votes is that people are meant to only vote for the things most important to them, instead of essentially "liking" everything that takes their fancy. You can move a vote once you've placed it, right? 10 only isnt enough because, as said above, there's no turnover in issues. Nothing is getting completed or closed for people to re-assign their votes to new things, and there's so many current active issues that 10 votes doesn't go far at all and actually wouldn't represent meaningful data (unless perhaps I picked one thing and put all of my votes on it).

 

I didn't make my point clear enough. Wiki is about facts, while Unity feedback is about wishes. The reason why people contribute and how they can do it is completely different and thus it is extremely vague to compare them.
But it doesn't matter, as it is not the point of the discussion.

 

i'm 100% for better mono/gc

or for game that run on windows allow us totally to replace mono with microsoft's .net for both unity3d editor on windows and for unity3d games on windows.

 

Now we are approaching the one year anniversary of UT requesting a list of API functions causing unnecessary allocations, I'd like to know where UT are at with implementing a cleaner API?

I'd appreciate either Joachim or Lucas providing us with an update.

 

I second that.

In my A* Pathfinding Project I have very large amounts of code written only to circumvent the hiccups the not so good GC Unity uses at the moment is causing.
Cool features are nice, but I would choose a good GC over it any day.
If we didn't have to work so hard to get smooth framerates, more work could be spent on actually developing the game and adding features to it instead of fighting the GC.

 

I tried to send a tweet to Lucas this morning. I hope this will bring his attention on this topic with a lot of good news .
Anyway, there is no update planned for the current GC @ Unity based on "new Mono GC" (if SGen could be called new). They try solutions internally if I'm not mistaken.

 

Out of curiosity, does anyone here have any experience with garbage collection in other game environments such as unrealscript in the Unreal Engine or C# on the Windows platforms? A quick google of UDK garbage collection performance yields nothing useful.

Is garbage collection just a problem in Unity or is any managed environment going to have the same issue?

 

I have some experience with garbage collection in early versions of XNA. I don't know what it's like in XNA 3 and 4, but XNA 1 and 2 is where I picked up my habits of not allocating during periods where I wanted a stable frame rate. It's also worth noting that while it wasn't a big deal on PC, it was a huge issue on the Xbox if not appropriately managed. My first XNA game on the 360 went from a few FPS to 60FPS just by pre-allocating everything into pools, despite running at a flat 60 on PC either way.

 

I'm just wondering if a GC that runs efficiently enough for games exists anywhere. I haven't heard of this being an issue with unrealscript but maybe it's just not common knowledge.

 

Unrealscript isn't as generic as Mono, so it could have underlying assumptions built in that make it a non-issue or which allow their GC to work much faster. Mono/.NET is designed as a general purpose language, UnrealScript is designed specifically for that engine by someone specialized in making scripting runtimes for real-time systems.

 

So the question is - considering that UT are no longer upgrading to the latest mono, are they working on their own optimized version? And if so, when would we expect to see it?

I don't expect an answer to this question, but it's something to ponder.

 

Yeah they work on internal solution but the release date is unplanned, if something is released one day.