Corrupted stack traces with il2cpp on iOS

Hey

We have an automated crash reporting system as well as report various errors that occur in the game (not crashes).

On iOS, many of these are reported with a stack trace that just doesn't make any sense.
We know that there used to be issues regarding that in the past, but i believe these were already resolved.

We are using Unity 5.3.4p1, which is relatively up to date.

Anyone else experiences such issues ?

BTW, we get the errors + stack traces using Unity's event 'logMessageReceived':

Code (CSharp):

1. Application.logMessageReceived     += ReportError;

Code (CSharp):

1. private void ReportError(string logString, string stackTrace, LogType type)
2. {
3.     // use the stackTrace string when reporting the error
4. }

 

Hi Josh,

The problem is that these issues are pretty random, we're only getting them from real world players, and can't always reproduce them in house.

Also, I do have to get permissions from my employers in order to share our project with you guys (i have tried to get that in the past but did not get it unfortunately).

I can share a few scenarios of stack traces that can't be right:

NRE with no stack trace at all:
NullReferenceException: A null value was found where an object instance was required

This is the code that is reported to throw the exception:

Code (CSharp):

1. private bool canSelectItem = true;
2.  
3. /// <summary>
4. /// A flag that determines whether a leaderboard item can currently be selected (clicked on).
5. /// </summary>
6. public bool CanSelectItem
7. {
8.     get { return canSelectItem; }
9.     set { canSelectItem = value; }
10. }

I need to look for a few other examples, we had others, It looks like it's mostly NullReferenceExceptions.

 

Thanks Josh! i'll upgrade and update this thread in case the issue was fixed (or not.... )

 

No signs of 5.3.5 or p1? has there been a delay?

 

Thanks you was correct, official has been released. Any internal news on the first patch?

 

We just started using 5.3.5p1, still getting corrupt stack traces, here's an example:

 

My company cannot share the project, not without protecting themselves in some legal way (it's not up to me, sorry...).
I will see what the latest decision is.

In the meantime, please note that it seems all stack traces that we report now being with the same method calls (which are clearly wrong), here are 2 examples:

Fyber :: Request failed due to: An error happened while trying to retrieve ads
MALog:ReportError(String, String, LogType)
FyberPlugin.FyberCallbacksManagerrocessRequestCallback(NativeMessage)
System.Collections.Generic.ICollection`1:get_Count()\n
System.Collections.Generic.ICollection`1:get_Count()
System.Collections.Generic.ICollection`1:get_Count()

UnityException: SetAnimationImpactTimes : No events in shield animation
ShieldMainAnimation.SetAnimationImpactTimes (Int32 shieldCount)
ShieldMainAnimation.PlayAnimation (Int32 amountOfShields)
MainManager+<playShieldSequenceInternalIenum>c__Iterator92.MoveNext ()
System.Collections.Generic.ICollection`1[T].get_Count ()
System.Collections.Generic.ICollection`1[T].get_Count ()
System.Collections.Generic.ICollection`1[T].get_Count ()
System.Collections.Generic.ICollection`1:get_Count()

Not sure there's much you can do without a project though...

 

I will check that; Other info that (may?) be useful: we did not yet upgrade our project from 5.3.4p1.

The way i do it, is i flip the switch in Cloud build to set a different version of the engine. I tried both 5.3.5p1 and 5.3.5p4 today, both exhibit this issue. I will try to see if there's any Xcode log in Unity cloud build and will report it if i find anything.

 

I found this in the log:

Is this related? (i will send you the full log in a PM if that is possible)

 

Just did thanks !!

 

I'm seeing a slightly different issue, but am wondering if it's somehow related. We've developing a game for iOS and Android. We use an in-game logger for our development builds called Lunar Mobile Console, and it shows us the Unity stack trace for all Debug.Log statements. We also publish our dev builds through HockeyApp, so we get to see the exceptions there as well. In LMC, in iOS, I'm seeing stack traces on normal Debug.Log statements that look like this:

However, in the Unity editor, the same log message looks like this:

There are two things of interest here: First, is it possible to get line numbers like we get in-editor? It would make tracking down exceptions so much easier. Second, why has the iOS build injected spurious calls to various methods (e.g., ThirdParty.MD5.MD5Managed:HashFinal and IAnalyticsFieldNameAdapter:Adapt)? The analytics field name adapter is literally a 4-line interface with 1 method that simply adapts typed fields into their string counterparts readable by our analytics system. Furthermore, it isn't even in the call chain for this particular log statement, so I'm not sure why it's even showing up.

We're running Unity 5.3.5p2 against Xcode 7.3.1 using the Debug configuration with both Development Build and Script Debugging selected.

Update: I just noticed this crash come in from HockeyApp. It's sort of ridiculous:

The Amazon method doesn't cause that error to be logged, DC.Common.Tuning.GetDefById does. Why is the Amazon method there? And why are there now EIGHT mentions of the (not-at-all-in-the-actual-call-stack) IAnalyticsFieldNameAdapter#Adapt method? Hoping this is a known/fixed issue in 5.4...

 

Are you building your game from cloud build? if you are, it may be similar to the issue we had. The stack traces could not be relied on (see my examples above). If you are using cloud build you should use 5.3.5p7 or higher this should solve that issue.

If you are not using cloud build, you should check out newer patch releases as they *may* resolve other corrupted stack trace issues, though i am not sure.

Regarding line numbers - the editor runs your C# code so you can get line numbers, but as the code is transformed into C++ on iOS (using il2cpp) this information is lost... the code that is executed is no longer c# and these line numbers are no longer relevant ...

 

Thanks for the reply, liortal.

We're actually building with Jenkins. I'd like to try Cloud Build, but I'm a bit worried it won't work for our circumstances (we have both client and server solutions with multiple shared projects and it just seems easier to use one system than two).

I haven't tried 5.3.5p7 yet, but I've been wary of upgrading beyond p2 because there's an unresolved issue I've had with p3 - p6 that seems have been deprioritized at this point. It's a hard crash almost immediately after entering the game (due to something in Unity's implementation of UnityWebRequest) so I'm kind of stuck between a rock and a hard place. I'll try p7 tomorrow morning, though, and will see if it solves anything -- or at least see if it doesn't cause our game to crash anymore.

Regarding the line numbers, that totally makes sense, I was just sorta grasping at straws.

 

Could you elaborate on the console logger you're using? what is it for ? and would you recommend it ?

 

I'd actually very much recommend it -- it's been invaluable for showing us in-game errors and general logs. It's called Lunar Mobile Console (they also offer something called Lunar Plugin that I haven't tried -- it's more akin to the old Quake console, if you're old enough to remember futzing around with that, heh).

Basically, you two-finger-swipe down from the top of your device and the console shows up. Think of it like Unity's console, with Log, LogWarning, and LogError messages displayed therein. You can tap any given entry to show its full message and any relevant stack trace (or in my case, something mostly relevant), filter by info/debug/error types, filter, and even email logs if enabled.

A few great things is that it is entirely self-contained within a single folder, so you don't have to pick apart your project if you decide not to use it anymore; it works on both iOS and Android; it's open source; it also has batch mode support so you can disable it for production builds with one executeMethod call. It's pretty actively developed, too, and the guy responds to questions really quickly on the forum thread.

 

Here's a screenshot of my build logs, Josh:



The Run Script step ran just fine, however I can't find any references to MapFileParser when I expand the highlighted step. Maybe that's an issue? I know I've seen a reference to that before while digging around various files, but I can't remember where.

About the line numbers, that's understandable. Would be nice to get something like what CoffeeScript and Sass have (source maps) even if performance went out the window, but I know C++ and JS/CSS are entirely separate beasts, heh.

 

I just did a find in the entire build directory for those files and they didn't come up. However, I also realized where I'd seen the MapFileParser... in the root build directory itself, heh. Here's the terminal output:



So it's definitely there, and I can see in the shell script that it's calling into the MapFileParser executable, but maybe the script just isn't getting run by the build process. A reference to
"$PROJECT_DIR/MapFileParser.sh" is also listed in the "Run Script" section of the "Build Phases" tab of my project, so I'm not quite sure what's going on there. Do you know of a way to check its output? Since it's not showing up in the build log, I'm kind of at a loss.

 

I'm not familiar with how to do that. I built an archive, right-clicked the resulting archive's entry in the "Archives" window and chose "Show in Finder." That took me to a file called Unity-iPhone 7-14-16, 11.45 AM.xcarchive in Finder. I opened a Terminal tab there because I figured it was actually a directory, and inside I found 4 directories (BCSymbolMaps, Products, SCMBlueprint, dSYMs) and 1 file (Info.plist). I don't see a log directory anywhere, and the BCSymbolMaps directory only has one file, D9DDC238-4158-3365-BC87-FF4CDDD024B1.bcsymbolmap. Maybe I did something wrong or am looking in the wrong place. I'm not terribly familiar with Xcode.

I added a few echo statements to the MapFileParser.sh file and here's what I've gathered:

ARCHS: armv7
PROJECT_DIR: /Users/adam/dev/games/builds/iOS
TARGET_TEMP_DIR: /Users/adam/Library/Developer/Xcode/DerivedData/Unity-iPhone-bclunkiqbvdildbkgpidyesqeadb/Build/Intermediates/Unity-iPhone.build/Debug-iphoneos/Unity-iPhone.build
PRODUCT_NAME: dungeon
CURRENT_VARIANT: normal
CONFIGURATION_BUILD_DIR: /Users/adam/Library/Developer/Xcode/DerivedData/Unity-iPhone-bclunkiqbvdildbkgpidyesqeadb/Build/Products/Debug-iphoneos

Inside the TARGET_TEMP_DIR, I do see a dungeon-LinkMap-normal-armv7.txt file, so I'm assuming MapFileParser has done some of its job, but none of the SymbolMap-* files exist (again, this might be the wrong directory).

 

OK so strangely, I can see that file in my local machine when I navigate to the archive directory's Products/Debug-iphoneos/dungeon.app/Data/Managed directory. I can also see it locally in Xcode's DerivedData directory (similar to the path of the TARGET_TEMP_DIR I mentioned above). However, when I go to my build machine and go into Xcode's DerivedData directory for the target build, SymbolMap-32 doesn't exist there.

The only difference between what I'm doing locally and what we're doing on the build machine is that I'm kicking off local builds through Xcode's UI under Product -> Build, whereas Jenkins is building with the xcodebuild command line utility. They both use the same configuration (Debug). The xcodebuild command line doesn't seem suspect to me:

/usr/bin/xcodebuild -target Unity-iPhone -configuration Debug build CONFIGURATION_BUILD_DIR=/Users/MacMini/jenkins/workspace/dungeon-ios/Client/Builds/iOS/build "CODE_SIGN_IDENTITY=iPhone Distribution" CODE_SIGN_RESOURCE_RULES_PATH=$(SDKROOT)/ResourceRules.plist DEBUG_INFORMATION_FORMAT=dwarf-with-dsym

Any ideas? The only difference I can tell is that the build machine has Xcode 7.2.1 while I have 7.3.1 locally. I can try to upgrade the build machine to the newer version if you think it would help.

 

OK, I just upgraded our build machine to 5.3.5p8 and I'm now seeing the SymbolMap-32 file in the Data/Managed directory. Thanks for bearing with me through this. I'll keep you updated as to whether or not this fixes my incorrect stack trace issues.

 

Hi Josh,
Seem the stack traces are still corrupt, at least on Android.
Tested on 5.4.2p2
This is a stack trace on Il2CPP Android:

Code (CSharp):

1. Ex [HTTPRequest]: CallCallback - Message: 1: A null value was found where an object instance was required.   at UnityEngine.Purchasing.Default.WinProductDescription..ctor (System.String id, System.String price, System.String title, System.String description, System.String isoCode, Decimal priceD, System.String receipt, System.String transactionId, Boolean consumable) [0x00000] in <filename unknown>:0
2.                                                                 at UnityEngine.Purchasing.UnityNativePurchasingCallback.EndInvoke (IAsyncResult result) [0x00000] in <filename unknown>:0
3.                                                                 at UnityEngine.Purchasing.UnityNativePurchasingCallback.EndInvoke (IAsyncResult result) [0x00000] in <filename unknown>:0
4.                                                                 at UnityEngine.Purchasing.UnityNativePurchasingCallback.EndInvoke (IAsyncResult result) [0x00000] in <filename unknown>:0
5.                                                                 at Burando.Network.WebMessages.OnRequestFinished (BestHTTP.HTTPRequest request, BestHTTP.HTTPResponse response) [0x00000] in <filename unknown>:0
6.                                                                 at BestHTTP.HTTPRequest.CallCallback () [0x00000] in <filename unknown>:0
7.                                                                 at BestHTTP.Con

And this on mono Android:

Code (CSharp):

1.  Ex [HTTPRequest]: CallCallback - Message: 1: Object reference not set to an instance of an object   at Burando.Network.WebMessages.OnRequestFinished (BestHTTP.HTTPRequest request, BestHTTP.HTTPResponse response) [0x00000] in <filename unknown>:0
2.                                                                 at BestHTTP.HTTPRequest.CallCallback () [0x00000] in <filename unknown>:0   StackTrace:   at Burando.Network.WebMessages.OnRequestFinished (BestHTTP.HTTPRequest request, BestHTTP.HTTPResponse response) [0x00000] in <filename unknown>:0
3.                                                                 at BestHTTP.HTTPRequest.CallCallback () [0x00000] in <filename unknown>:0
4.                                                              
5.                                                               (Filename: ./artifacts/generated/common/runtime/UnityEngineDebugBindings.gen.cpp Line: 42)

IL2CPP thinks it is coming from Unity IAP which it isn't in reality.

 

I am using an "asset store" asset(besthttp), what's the policy in this case?
Can I include it?

 

Luckily i was able to isolate it. Case 846956.
By the way I saw it is now caching object files, so the second time is definitively faster.
Could it be that now the first time is much slower however or was my impression? Or maybe besthttp is very heavy to compile, don't know.
And in what situations it does a clean and refresh all objs?So I know what to avoid.