Utilizing crash.dmp files

This is something I've been wanting to figure out for a long time. It's unfortunately common to get player crashes in which the stack only contains internal engine methods and no user code. When these crashes are not readily reproducible, it becomes a nightmare for developers to find and fix the cause. In cases like this, there is effectively no guidance from Unity on how to debug these issues. I can find occasional questions on this topic popping up over the last five years, but never any real information.

I've recently realized the .dmp file is a typical Windows minidump. They can be opened with Visual Studio or WinDbg and symbolicated if the appropriate .pdb files exist. This is a nice step in the right direction, but atm even this is falling short of actually being useful. I have a lot of questions about dealing with engine crashes that I'm hoping to get some insight into in this thread.

1. How does Unity expect us to deal with crashes when there's no user code in the stack? Over the past year we've had on the order of a dozen crashes of this type and troubleshooting them has added up to a significant amount of time that could have been spent developing new content. These are almost always from subtle engine bugs (e.g. enabling a MeshCollider when Time.timeScale == 0, running out of DCB command buffer space, or the most recent which seems to be from checking "Prebake Collision Meshes" in the Player Settings). The only response I've seen from Unity recommended doing a binary search through your project by disabling chunks of content until the crash stops and narrowing in on it that way. But this is only even valid for readily reproducible crashes, which many are not.
   
   
2. Is there any way for us to react to crashes as they happen? Something akin to the AppDomain.UnhandledException event? Currently, our only option is to log copious amounts of data and hope we can deduce the state of the game during the crash later on. Since we don't know where a crash is going to occur, the only reliable way for that to work is to log the entire state of the application. It would be much more efficient if we could simply react during a crash. I assume the engine already does this since it's able to flush the log, add a stack trace, and create a crash report. If developers are given a way to react as well, we can keep a rolling buffer of state changes and simply dump the buffer during a crash. This would be far more effective than attempting to log all state in an ongoing manner. Is it possible to add this functionality for us?
   
   
3. We need a way to log without a stack trace. Collecting a stack trace is incredibly slow and it means we can't log any more than a trivial amount information in a single frame without causing massive performance issues. This makes logging enough data to reconstruct application state during a crash infeasible, eliminating one of our only ways of debugging engine crashes. We need a performant way to log lots of data when necessary. As it stands, we're going to have to implement a custom or third party logging solution to be able to log enough information to maybe track down a crash. Is this something that can be added?
   
   
4. As far as I can tell there is no mono.pdb. This means even with a dump from a development build, we can't symbolicate the user code portion of the stack. The `output_log.txt` actually includes names over user code methods in the stack, so the lack of this pdb actually makes dumps less informative than the plain text log. I assume, if properly symbolicated, the dump should actually let us see parameter values, which would be a nice extra piece of the puzzle. I've attached a screenshot of as far as I've been able to get with the dump file so far. Is there currently any way we can get the mono.pdb so we have all symbols? If not, would you be willing to provide this in near future releases?
   
   
5. Is it possible to separate symbol generation from the development build option? If we can get to a point where dump files are actually useful, I'd like to be able to create a release build with symbols. I'll store the symbols locally and be able to at least inspect the stack when crashes are reported. I know symbols won't be completely accurate due to optimizations for release builds, but information is power, and I want everything I can get my hands on. Also, I don't want users to see "Development Build" in the game just because I want symbols.
   
   
6. The dumps currently produced are minidumps with no heap information. Do we have any control over this? Can we enable full dumps for end users (while in beta, at least)? If not, is it possible to provide this option in near future releases?

As a developer making more and more ambitious games, if I'm going to continue using a black box engine, being able to respond to and debug crashes effectively can make or break the platform. Some community outreach on this topic would go a long way in boosting my confidence in Unity as a viable engine for future projects.

 

Bump. Still hoping to get some actual information from Unity here.

 

Well, there's some difficulty here. Most of the user code is c# which is in the Mono VM at the moment. (in the future, il2cpp will change this). There's no existing easy way to forensically extract the VM stack of user code at the moment from us. A quick search on google doesn't provide any SOS.dll windbg style help for Mono as there is in the MS .NET framework yet.
Though, if you're live debugging, you can try out the steps offered here.

Of possible use is the Application.LogCallback (answers).

Something is in the works...

There is, though it's renamed to mono_embed.pdb ( or was, I was trying to fix this ).
Alternately, we've been pushing the roll-up builds pdbs (not yet patches, but that's coming) to a public symbol store since 5.0: http://symbolserver.unity3d.com/ (requires VS2012 or later, or windbg). I guess i should really blog about this and add to the manual.
Note: this has been manually maintained right now which is why it's a little spotty on versions. It's getting incorporated into our publish scripts to be more consistent. I was mainly waiting on that to be more forthcoming on it.

Symbols aren't built on your side at the moment (well, until il2cpp I guess). We build the symbols that accompany the player. We already build the release player with symbols and they should be up on the symbol store listed above.

Are you talking about the ones generated with our bug reporter? Full dumps are pretty big.
I'd alternately suggest registering for Windows Error Reporting (WER) from http://sysdev.microsoft.com/. You'll need to get a cert and sign the exe. (Right now we do sign it, so you'd have to strip our cert. There are tools out there to do this). On WER, you can request full heap dumps at times, and for Win 10 previews they're right now always full heap for debugging purposes right now.

You're focused pretty specifically on Windows and we have to consider the other platforms too. However this is something we're working on improving. I was waiting specifically on having a more reliable delivery of pdbs before going too far with announcements.

 

First of all, thank you very much for taking the time to respond so thoroughly. There's some great information here that I'll be able to make use of (namely, the symbol server, mono pdb, and WER).

There may be a bit of confusion here. I'm asking about how to deal with crashes that don't directly involve user code. Crashes like this one (one of our current major issues)

00CA47A0 (RoadRedemptionEarlyAccess) RenderTexture::ExtractImage
01130773 (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
011B3037 (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
011B4366 (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
011B5A21 (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
011AE18B (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
0113024E (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
0112D857 (RoadRedemptionEarlyAccess) Texture::GetMipMapBias
013224D9 (RoadRedemptionEarlyAccess) RenderTexture::GetClassIDStatic
00CA5093 (RoadRedemptionEarlyAccess) RenderTexture::ExtractImage
00CAAD72 (RoadRedemptionEarlyAccess) RenderTexture::ExtractImage
00B2688F (RoadRedemptionEarlyAccess) UserList::AddUser
00BBDF61 (RoadRedemptionEarlyAccess) PlayerMainWndProc
00BBFBF0 (RoadRedemptionEarlyAccess) PlayerWinMain
00EF7958 (RoadRedemptionEarlyAccess) PlatformThread::Exit
00F2E98F (RoadRedemptionEarlyAccess) PlatformThread::Exit
75CC337A (kernel32) BaseThreadInitThunk
771C92E2 (ntdll) RtlInitializeExceptionChain
771C92B5 (ntdll) RtlInitializeExceptionChain

This crash is likely caused indirectly by our code, by putting Unity into a bad state and it causing it to choke when doing its normal, internal stuff each frame. But no user code is even running at this point, so I'm not sure there's a relevant Mono stack at all. My question is how can we even start to track something like this down when it's not reliably reproducible? How can we get more information from the engine?

Is this reliably called on any possible form of engine crash? This is not something I had even considered in those circumstances, to be honest. It looks like crashes raise logMessageReceived with LogType.Error, so I assume we're going to have to be careful to sift through less critical errors like "cleaning the mesh failed", but that gives a hook to dump game state

The symbol store looks very promising and will likely address this problem.

Yes, those are the dumps I'm referring to. I realize they are quite large, but at this point there simply appears to be no other way to efficiently narrow down the cause of engine crashes. I'll be looking into WER today, but the ability to decide that functionality at runtime from within Unity itself would be great.

Admittedly yes, as the majority of our players are on Windows and by extension we have the most information on that platform. And unless the crash is platform specific, we only need to debug and fix it on one platform to fix it for everyone.

 

Ah. My misunderstanding. Unfortunately, without source code you're right that this is pretty much impossible to address. The request would be to submit a bug, as it'd be undesirable behavior from our part too. To expedite bug attention, we do have our premium support. However, otherwise post the bug number along with the problem in the appropriate forum and often a dev may come help.

Though, in the example given - that stack looks really really wrong. Depending on the crash type, I'd wager that stack isn't actually accurate. (Which makes it all the more harder to debug for us....repros help!)

Honestly, I just did a little digging to find that myself, but I'm not particularly familiar with that hook. I believe it'll skip certain exceptions but the major ones should be there.

Honestly, having heap without source is pretty limiting on debugging. Looking at where we generate minidumps, we actually only do it from our bug reporter from what I'm finding, which is usually with the editor or web player (IIRC). So, I'm not sure which dmp files you're talking about?
In any case per my previous post registering for WER is currently one of the cleanest ways of getting minidumps and selectively full dumps from users.

 

This is absolutely terrifying to me. I have a black box that crashes and gives me effectively no information to work with.

We've invested a lot of time and effort into learning how to push Unity to make a really good product, and on our first larger scale game we begin to realize that if anything goes wrong, it's going to take weeks of guessing and checking to even have a shot at fixing it. This is the type of thing that prompts discussions on changing engines.

I'm sure you guys don't want crashes, but the truth is they aren't going away. There are always going to be strange edge cases where they occur, or places where you simply can't prevent them. A few months back I reported a crash (on PS4 caused by an old version of Marmoset shaders, IIRC) and was told 'we can't check for everything because it would hurt performance, you're stuck with this' (paraphrased).

If we had a reliable repro case, we wouldn't be in suck a pickle to begin with

If my guess is correct, and it's something we are doing to put Unity in a bad state being able to inspect the full state of the application can help us easily find where we're doing something goofy. In the current case, we're seeing a lot of these guys

Infinity or nan floating point numbers appear when calculating matrices for collider.​

and suspect it might be related (we guessed the stack might be incorrect as well). With heap, we could dig around and at least see where those NaNs/Infinities are coming from since the engine doesn't tell us.

Any time the engine hard crashes on Windows you get a little popup saying the game crashed. A folder is created next to the executable with a timestamp as the name which contains output_log.txt, error.log and crash.dmp. This is easily seen by creating a build that calls Application.CommitSuicide(0).

We need a better way to debug these kinds of (fairly common) crashes. Paying for source, paying for premium support, or posting on the forum and hoping are not very good solutions for indie devs struggling to make their first big game with this engine.

 

Our goal is to make the game making accessible to everyone. So crashes need to be pushed back to warning and errors so that people can fix things without needing the technical ability of post-mortem crash debugging. While we want to also facilitate the more technical customers, we need to also enable the non-technical customers.

May I point out also that you're running on a black box of an operating system too? All Windows developers encounter similar bits where stacks land squarely in OS modules that they don't have source for. The same can happen for 3rd party libraries that you may integrate.

Indeed, it's a trade off that's made regularly for games, perf vs. stability. If the data/inputs can be sanitized, then checks can be removed. Though conversely, it could be considered for doing more data checking during the build phase. Sorry for the situation you had, but at least you knew the cause. That allows alternate solutions at least.

Maybe...forensic analysis does not always lead to a proper determination of cause. Stack corruptions/memory scribbles/etc. leave tell tale signs in the dumps, but not always a trail to follow back to the original cause.

In the case of non-useful messaging, that too could be a bug for us to improve. Seems that message is more useful for us, but not you without specifying the object/motion. I'll bug that for the physics folk.

Ah, ok. I overlooked that the crash handling is linked into clients. Again, not sure of heap contents being of particular value to you, but certainly we can consider the feedback of configuring for full dumps. I'll add a bug for that.

Agreed that we should make it easier to debug crashes, it's on our list of things to improve. That's why we're taking the steps to do a sym store and looking to expand other areas to improve self-analysis. However, it's not a cure all, hence we have the community as a first stop, then our premium support. The premium support is definitely not a pathway for moneymaking for us. Let me just emphasize it is not free because offering support has definitive costs for us. Note that we have an indie level for premium support as well.

 

There's a single mono.pdb in Unity\Editor\ and there are 7 mono.dll. Using chkmatch,

The pdb matches the following dlls:
"Unity\Editor\Data\Mono\EmbedRuntime\mono.dll"
"Unity\Editor\Data\PlaybackEngines\windowsstandalonesupport\Variations\win64_development_mono\Data\Mono\mono.dll"
"Unity\Editor\Data\PlaybackEngines\windowsstandalonesupport\Variations\win64_nondevelopment_mono\Data\Mono\mono.dll"

But not these:
"Unity\Editor\Data\Mono\bin\mono.dll"
"Unity\Editor\Data\PlaybackEngines\windowsstandalonesupport\Variations\glesemulator\Data\Mono\mono.dll"
"Unity\Editor\Data\PlaybackEngines\windowsstandalonesupport\Variations\win32_development_mono\Data\Mono\mono.dll"
"Unity\Editor\Data\PlaybackEngines\windowsstandalonesupport\Variations\win32_nondevelopment_mono\Data\Mono\mono.dll"

This is while using the symbol store you linked. Luckily, we're making 64-bit builds, but there's another hiccup with those. For some reason, even though I can load symbols for mono.dll, the stack remains almost entirely unresolved. I've attached a screenshot (and a 32-bit one for comparison).

 

Alex,

I appreciate your support.

I work with Sycobob on the same project (Road Redemption), and this crash has already taken up two weeks of our studio's time, and I can't say we're any closer to solving it.

After days of testing, across multiple machines, we've only managed to reproduce it once in our office. Yet every day, we receive about a dozen crash dumps from customers who are experiencing it. This crash is costing us goodwill, review scores, and Steam refunds.

I've contacted Unity sales department about taking advantage of Premium Support, and I am waiting to hear back from them. I lament the fact that we will be spending even more money trying to solve this engine crash when we'd be capable of solving it ourselves, had we the proper information.

I've attached three crash .zips, from three separate users. Each .zip file contains the output_log.txt, error.log, and crash.dmp.

Each output_log.txt displays the mysterious "Infinity or nan floating point numbers appear when calculating matrices for collider." right before the crash.

Any help would be extremely appreciated. Thank you.

-Ian Fisch
Darkseas Games
1-504-289-4683

 

Ah yes. Because of the manual process I'm not bothering to do the 32 bit ones at the moment. Once we're automated it should be all there. Going to ask the deployment team to see about testing out the script with 5.1.1f1 which seems to be what you were using with the crash dumps.

 

Sorry for your frustrations, but I share your pain (and have before myself in games). Debugging can be plain hard. One team I was with took 5 months to root out a rather nasty bug in a 3rd party library that was ultimately our own doing of passing bad data in. We are working to improve the self-debugging scenarios, but they won't be arriving overnight. They'll take some time to do and iron out.

The right approach for the future is to create a bug for us via the bug reporter/issue tracker. Then post the follow up bug #. However, in by-passing our standard in-flow, I took a quick look with windbg. It's down in vehicle physics where a collider seems to be out to lunch as passed back by PhysX. Dunno if perhaps an object got deleted too quickly, as it's not a null value.

Could you please create an appropriate bug and submit? Then I can have it tossed over to the physics devs and/or the support folk can more clearly follow up.

 

Cool. The vast majority of our crash reports have been from 5.0.2f1, but we're in the process of upgrading to 5.1.1.

I realized I was mistaken when I said we were making 64-bit builds. In fact, we're making 32-bit ones. So we're still unable to symbolicate them fully. We have a couple crashes with user code in them that I'd really like to be able to check out. Any chance it's possible to throw the 5.0.2f1 x86 mono.pdb on the symbol server? The signature is 352655f5-83d0-48ba-ba03-4abf4de64639.

Regarding the 64-bit issue, it's not terribly important now, but do you have any idea why those aren't symbolicating properly?

 

This is golden! Without a symbol server a managed dump is mostly unreadable.

Btw, I use PSSCOR instead of SOS and it's awesome.

 

Alex, I posted a bug which can be found here: http://fogbugz.unity3d.com/default.asp?708101_qtfkqpe5c60idiva
I attached the same .zip files that I attached to this thread.

I appreciate you alerting the physics devs of our situation.


I was wondering if you could elaborate on this:

Thank you.

 

Here's a cut'n'paste from windbg with what I saw.
Note on the failing assembly, that it's moving from a memory location listed in esi.
It's not a zero value there, so it was passed some theoretical piece of memory that happens to be invalid.

VehicleWheelRaycastPreFilter is our function, but the next 8 functions in the stack were in PhysX.
So, either there's a plain bug percolating up to us, or as you said there's something being done wrong on your side, but you're not getting any alert to that fact. If it is your side, my very offhand shoot from the hip guess is some collider being removed in the same frame but not from the physics yet?

Code (CSharp):

1.  
2. RoadRedemptionEarlyAccess!VehicleWheelRaycastPreFilter
3. RoadRedemptionEarlyAccess!MultiQueryCallback<physx::PxRaycastHit>::invoke
4. RoadRedemptionEarlyAccess!processBucket<0>
5. RoadRedemptionEarlyAccess!stab<0>
6. RoadRedemptionEarlyAccess!physx::Sq::BucketPrunerCore::raycast
7. RoadRedemptionEarlyAccess!physx::Sq::AABBPruner::raycast
8. RoadRedemptionEarlyAccess!physx::NpSceneQueries::multiQuery<physx::PxRaycastHit>
9. RoadRedemptionEarlyAccess!physx::NpBatchQuery::execute
10. RoadRedemptionEarlyAccess!physx::PxVehicleUpdate::suspensionRaycasts
11. RoadRedemptionEarlyAccess!UpdateVehicles
12. RoadRedemptionEarlyAccess!PhysicsManager::FixedUpdate
13. RoadRedemptionEarlyAccess!PlayerLoop
14. RoadRedemptionEarlyAccess!MainMessageLoop
15. RoadRedemptionEarlyAccess!PlayerWinMain
16. RoadRedemptionEarlyAccess!WinMain
17. RoadRedemptionEarlyAccess!__tmainCRTStartup
18. WARNING: Stack unwind information not available. Following frames may be wrong.
19. kernel32!BaseThreadInitThunk
20. ntdll!__RtlUserThreadStart
21. ntdll!_RtlUserThreadStart
22. 0:000> .ecxr
23. eax=75636c61 ebx=00d3f070 ecx=69740000 edx=7fa51b9c esi=75636c61 edi=3f749508
24. eip=01632890 esp=00d3eb08 ebp=00d3eb14 iopl=0         nv up ei ng nz na pe cy
25. cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00210287
26. RoadRedemptionEarlyAccess!VehicleWheelRaycastPreFilter+0x30:
27. 01632890 8b16            mov     edx,dword ptr [esi]  ds:002b:75636c61=????????

That's about all the help I can offer. I'll defer the rest to the physics guys with the bug or the support channel if you choose to use it.

 

Interesting! Didn't know about that update replacement to SOS. Will have to check that out.

 

We're making 32-bit builds, and the mono.pdb isn't available yet, so we're still unable to symbolicate them fully. We have a couple crashes with user code in them that I'd really like to be able to check out. Any chance it's possible to throw the 5.0.2f1 x86 mono.pdb on the symbol server? The signature is 352655f5-83d0-48ba-ba03-4abf4de64639.

 

Thanks again for looking into this issue.

I'm not exactly sure how a collider could be "removed in the same frame but not from the physics yet". We destroy, disable, and deactivate colliders in our code, but how exactly do we make sure the physics engine is aware of this? Is this something we have any control of?

Also, would it be possible to see the source for VehicleWheelRaycastPreFilter? Maybe that might help us get to the bottom of this issue.

We also have a theory that this could be related to the vehicle's rigidbody collision detection being set to "continuous", but without being able to reliably reproduce this crash, we can't test this theory internally. The game benefits a lot from the bike rigidbodies being set to "continuous" so we're loathe to release a public build with this setting disabled, just to test this one theory.

Thanks so much for all your help.

bug link: http://fogbugz.unity3d.com/default.asp?708101_qtfkqpe5c60idiva

 

I was also wondering if there's a way I can update the fogbugz bug report.

http://fogbugz.unity3d.com/default.asp?708101_qtfkqpe5c60idiva

Specifically, I'd like to mention that we do indeed enable and disable wheel colliders quite often in our code.

For instance, if a motorcycle hits an obstacle that throws the rider and cycle into the air, we disable the cycle's wheel colliders to ensure a more crunch when it hits the ground. 2 seconds later, we re-enable that motorcycle's wheel colliders when the rider respawns.

I'm guessing frequent enabling and disabling of wheel colliders is pretty rare in a Unity engine game, and I wouldn't be shocked to hear that this is related to our NaN issue.

 

Hey there,

I stumbled upon this thread because we too are having some (at the same time luckily and sadly not too common) crashes that seem to be related to "Infinity or nan floating point numbers appear when calculating matrices for collider."

And this is exactly where I would like to chime in: This error is just one of the examples where Unity does log some error but it's plain useless without getting a stacktrace or any other information on where it occured. Another example that I know from top of my had that I often got is regarding Tex2D.SetPixels and source array not being of the proper size. Even if the infinity error might not *directly* be a followup on a code line (though I think it probably is ) the SetPixels one definitely is so in such cases *always* getting a stacktrace would help a lot.
TL;DR: If Unity was *always* printing a stacktrace (if it's enabled in the build) on errors (that are already logged anyway) I suppose a lot of errors could quite easily be fixed by the game devs

Kind regards,
Chris

 

Here's the email I got from Unity after posting this bug:

Could it be the case where WheelCollider.GetGroundHit or WheelCollider.GetWorldPose are being called from inside of Update()?

Those functions access PhysX caches directly currently (and PhysX Vehicle cache is only validated during FixedUpdate), and can easily access broken data when called from Update() as they get PxShape* and try to find out the corresponding Collider*. This needs to be addressed surely, but in the meantime could you check if that may be the case in your project?

We were calling these functions a lot in Update() in order to position the visual tire meshes correctly. We moved the calls to FixedUpdate() in today's build and no crash reports yet, but too early to tell.

 

Just had the same problem: Installing the 32bit Unity Editor and using the mono.pdb from the editor there worked for us

 

This was the most informative post on crash reporting and debugging strategies ... thanks for Sycobob and Alex for writing such well thought out and fully detailed questions and answers. I to am working on a larger project and would love to see Unity expand the crash reporting for all platforms so that we can collect both native and managed issues and troubleshoot them in some sane way.

Sycobob ... I'm dying to know the result, did you find the bug?

 

I have not, unfortunately. We've just started releasing updates again for the first time since we found this issue. We were hopeful that 5.1.3p3 would address the crashes (there was a patch note in 5.1.3p1 that mentioned fixing physics crashes linked to 3 items in the Issue Tracker). Alas, there's a steady stream of crashes coming from the current build. This is all happening within the last week or so. This week I suppose I'll figure out how we're going to deal with this. That's probably going to include combing through more logs, testing 5.2, testing 64 bit builds, and attempting to follow up the bug report. Oh joy. We did have a 64 bit build up briefly due a mistake in the build process and interestingly enough, we never received any crash reports from that version. That's one possible path forward.

The symbol server mentioned above is awesome, but it's only been updated a couple of times since I've started watching it, so it's not very reliable at this point. No PDBs for patch versions, which we're often forced to use. And it's a pain to even check the PDBs that are there to see which version of the engine they are for. You have to compare GUIDs in them using something like chkmatch (might be an easier way, I haven't looked for alternatives).

 

Sorry, we're still ironing out the automated mechanisms to update the symbol server. It hasn't caught up to that, and I've been swamped. Though, once it's done all the patches and future versions should be covered.

 

Thanks for the update Alex. Do you have a ballpark estimate of how long it might be before the automation is in place? I won't hold you to it, I promise.

 

Sycobob

I was wondering how you're receiving crash reports from your users? After looking at your Steam community, I would guess that you're receiving .dmp files who manually due to instructions from this post: http://steamcommunity.com/app/300380/discussions/0/594820656459543975/

Is that correct? I was wondering how and when the crash.dmp, error.log, and output_log.txt your users are sending you are enabled. Is it by chance by setting a registry value such as decscribe here in the Unity docs?
http://docs.unity3d.com/Manual/WindowsDebugging.html

I tried adding the undocumented (and I realize therefore I really shouldnt'd depend on it) Application.CommitSuicide(1) to my own game to see if I could test .dmp files getting generated, but it doesn't appear to do anything for my Windows desktop build and the game keeps running along without issue ... so I was wondering if the registery chagnes are required or if there some other mechanism that is generating these for your users and how you're testing it?

Also, do you receive any other crash reports from users somehow, whether that be from a managed crash or a native crash? The only tool that I can seem to find is the capturing of a managed crash using something like:

System.AppDomain.CurrentDomain.UnhandledException += OnHandleUnresolvedException;
Application.logMessageReceived += OnHandleLogCallback;

And then I would have to upload those callstacks to a server that can keep track of it (or something like HockeyApp or the numerous other crash reporter services).

I'm about to start my own alpha and beta programs for Rogue Wizards on Steam (http://store.steampowered.com/app/392260/), but would like to have as much crash reporting in place before inviting people into the alpha and there doesn't seem to be a lot of support from Unity in this area, which is what brought me to this thread.

 

We actually search the data folder for crash reports that were generated from previous runs, upload them to a server, then delete them. People also email crash reports occasionally, but we rely almost exclusively on the auto-upload mechanism.

I don't recall needing to set anything up to generate crash reports. I believe this is the default Unity behavior. If you are not seeing them, my first guess would be that something about your configuration is disabling them. The registry value you mention can be used to get a *second* copy of the crash report, and it will affect any application that crashes on your machine, not just your game.

I use Application.CommitSuicide as well, actually. Since our "analytics" consist solely of collecting crash reports, I need any abnormal termination to lead to a crash. Using an argument of 0 causes a crash and generates a crash report. 1 causes a Fatal Error to be logged but the application chugs along. 2 causes the application to stop responding and Windows prompts to close it. No crash report is generated.

I don't think System.AppDomain.CurrentDomain.UnhandledException works. Application.logMessageReceived is only helpful for exceptions, not crashes.

If you're about to go into public testing, here's what I'd recommend:

• Do your best to ensure your game doesn't implode/go into unknown state when a problem occurs. This means attempting to handle all possible failure cases. Since it's kinda ridiculous to check for every possible case in every method try to handle state checking at the highest level you can, then just rely on it being correct at lower levels where you know the checks have already taken place. Don't allow exceptions to propagate up. Exceptions cause the callstack to abort all the way up until they are caught. This leaves you in an unknown state. Instead, return a bool or something handle failures appropriately.
  
  
• Use Application.logMessageReceived to log all unhandled exceptions and upload them to a server.
  
  
• Write a tool to auto-upload crash reports to a server.
  
  
• Consider writing a tool to allow users to easily report bugs from inside the game. e.g. press tilde to pull up a menu that allows users to type a short message and attach a screen shot.
  
  
• Unity does have some new infrastructure for collecting errors, but I don't know much about it yet: https://perf.cloud.unity3d.com/landing/ It's worth looking into.

For example, in our game the whole initialization chain for the application, in each individual scene, and for actual gameplay starts from a single object (we don't rely on Awake/OnEnable/Start for important stuff). All the init methods return a bool. If something fails we handle it immediately, whether that means retrying something, reloading the scene, or aborting back to the main menu. If something fails in a way that we know is irrecoverable, we throw a custom exception that gets caught in that top level object I mentioned above. That object uses Application.CommitSuicide to kill the game and we just hope the player runs the game again later so we get the crash report.

It's certainly far from ideal, but at least it's something. In the future we'll definitely be uploading all exceptions (we did this on another project and it worked wonderfully) and looking into Unity's performance reporting.

 

It's in progress, so perhaps in the next month or two? So it _may_ apply for 5.3 and forward. Not sure what will work on the older versions yet though....

 

Interesting suggestions. We've been building that last item noted - the game performance site to address some aspects of all this. To start, it really only handles managed exceptions from a handful of platforms. We'd like to expand that out to further reporting like native crashes as well, but its a work in-progress.

 

do you generate symbol information for your android players and if so, where can I get them?

 

can you please add the mono.pdb, 32 bit version, for 5.3? The folder should be http://symbolserver.unity3d.com//mono.pdb/F2C8DA40ABC24B4FB462F7FB70D949571/mono.pdb

 

it doesn't work with 5.2 either...am I doing something wrong?!
http://symbolserver.unity3d.com//mono.pdb/C0330A3822AB4DB6AE280E69E36A2B9F1/mono.pdb not found

I'd like to use mono_pmip, but I am not able to load the correct mono.pdb file. Would it work if I debug the editor instead of the client (as far as I understood it won't)

 

Thanks for the discussion so far. I also found it super helpful.

To contribute, here are some clarifying points on

UnityEngine.Application.CommitSuicide

and the different types of crashes:

• Application.CommitSuicide(0)

  is useful for simulating a crash that is caught gracefully by Unity - this doesn't log a Windows error event. (Thus I don't think WER would be useful for reporting these kind of crashes...?)

• Application.CommitSuicide(1)

  causes an error that Unity fully handles so it just ends up as a Debug.LogError entry. Unity and most existing error-handling solutions are aimed at this kind of error, and ideally the Unity engine pushes as many crashes as possible to warnings and errors.

• Application.CommitSuicide(2)

  is useful for simulating a native crash - this triggers an event in the Windows Event Log, but there is no Unity-generated crash report. (However, as mentioned, it's possible to configure Windows to generate full dumps on the local machine, and to configure WER.)

[EDIT:

UnityEngine.Application.CommitSuicide

is no longer available. I think it was removed from Unity 5.6 or possibly earlier. You can use

UnityEngine.Diagnostics.Utils.ForceCrash

instead.]

Also, here are some related notes on logging & debugging:

• Application.logMessageReceivedThreaded

  is likely preferred over

  Application.logMessagesReceived

  if you're using threads in your application

• System.AppDomain.CurrentDomain.UnhandledException

  is useful to for handling exceptions that occur outside of the Unity main thread

• System.Diagnostics.StackTrace

  can be useful to get a detailed stack trace in certain cases with release builds where the stack trace isn't logged by default.

 

I have packed a tool for handy analysis of Unity crash.dmp files.
https://www.nuget.org/packages/UnityCaptureTraces.x64
It is powered by MS cdb + some windbg scripting.
I tried it for couple of Unity crash dumps for now. It showed managed stacks, but surprisingly didn't show any managed exceptions at least for 2 different crash dumps, which I had.

 

Ivan, thanks for providing that script/wrapper.

I am also attempting to debug a usually-reproducible crash in 2017.3 with no luck in finding the culprit. Near the top of the CDB output I see:

And nothing else readable (to me) elsewhere. Is there a trick to getting the proper trace information in that output?

 

What is the state of debugging .dmp files in unity in 2019? This is the only relevant thread I've found on the topic but seems there hasn't been much news.

I've been trying windbg and Visual Studio to analyze a dmp, but in both cases, I haven't been able to get any managed stacks.

In windbg, for whatever reason I can't get SOS or sosex or any of the recommended tools for debugging .NET hangs to work (seemingly due to not being able to use the clr dll for whatever reason).

In visual studio, I have the symbols servers loading symbols but can't dig into mono-2.0-bdwgc. It says "mark.c" not found. And when I use unity's recommended command

 ?(char*)mono.dll!mono_pmip((void*)0x00000202e0906110)

it says 'Module "mono.dll" not found'.

@IvanShakhov I used your tool and ran it. It gave a little more info, so that's appreciated. But I think it has the same problem. It says

Code (csharp):

1. 0:000> .loadby sos mscorwks
2. Unable to find module 'mscorwks'
3. 0:000> .loadby sos clr
4. Unable to find module 'clr'

Despite me copying those .dlls from my v4.0.30319 folder into the C:\Windows\Microsoft.NET\Framework64 folder as indicated by robocopy.

Any updates would be appreciated

 

Any way to view crash.dmp on windows? going through hell here.

 

thanks, though it doesnt seem to give any more info on the bug than logs.