Re: OS X Crash Log Symbolication
Re: OS X Crash Log Symbolication
- Subject: Re: OS X Crash Log Symbolication
- From: Jerry Krinock <email@hidden>
- Date: Wed, 10 Jul 2013 17:10:05 -0700
On 2013 Jul 10, at 13:16, Wim Lewis <email@hidden> wrote:
> We have some scripts to parse the crash logs, find the build UUIDs and load addresses, find the relevant .dSYMs in our archives, and symbolicate everything using 'atos'. It's not too hard to do by hand, though of course if you have more than a few crash logs to deal with it's easier to automate it.
Once upon a time I found an app for symbolicating crash reports, but it didn't always work. There are corner cases that are difficult to cover. One problem is that I understand the tools use Spotlight (eeek!) to find .dSYM files. And then there's Apple's high rate of change.
Here is Apple's document which explains how to symbolicate…
https://developer.apple.com/library/mac/#technotes/tn2004/tn2123.html
But it has a last revision date of 2008-04-01. It still tells you to use gdb, and spends a lot of space on Power PC crashes. One time I tried substituting "lldb" for "gdb" but it didn't work. So, I'm still using gdb. I just filed a bug (14409079) asking Apple to update it.
Jerry
Until Apple updates TN2123, here is my "cheat sheet"…
****************************************************************
*** ALTERNATE WAY TO GET ACTUAL LOAD ADDRESS OF A FRAMEWORK ***
****************************************************************
Unfortunately, exception logs do not indicate the Actual Load Address of a framework, which is needed to symbolize addresses in it. But you can try this magic formula discovered by Nick Blievers. Say that you want to symbolize the calls to MyFramework in here:
2 CoreFoundation 0x00007fff907c234a -[NSArray arrayByAddingObject:] + 138
3 MyFramework 0x0000000100152704 MyFramework + 614850
4 MyFramework 0x0000000100152cd5 MyFramework + 616339
5 MyFramework 0x00000001000395db MyFramework + 202203
6 Foundation 0x00007fff92f34677 __NSThreadPerformPerform + 225
Find the *lowest* call in MyFramework, line 5 in this case. Subtract the second (decimal) number from the first (hex) number. In this example
0x00000001000395db - 202203 = 0x100008000
You should get a number that is a multiple of 0x1000. If so, this is the Actual Load Address (A) which you should offset addresses with when getting line info from the debugger.
********************************************************
*** STEPS TO SYMBOLIZE A CRASH DUMP OR EXCEPTION LOG ***
********************************************************
1) Move/copy both the crashing .app and the relevant.dSYM files into the same directory.
2) If the crash was in a framework, dynamic library, or bundle, you have position-independent code and must work out the *slide*. Otherwise, skip up to step 9 and use slide=0.
3a) If you have a crash dump, look in the *Binary Images* part of the crash log and find the framework, dynamic libray or bundle in which the crash occurred. Read the first column of the output. This is the address at which the __TEXT segment was loaded. We call it the *Actual Load Address* and denote it as *A*.
3b) If instead you have an exception log, like this:
2 CoreFoundation 0x00007fff907c234a -[NSArray arrayByAddingObject:] + 138
3 MyFramework 0x0000000100152704 MyFramework + 614850
4 MyFramework 0x0000000100152cd5 MyFramework + 616339
5 MyFramework 0x00000001000395db MyFramework + 202203
6 Foundation 0x00007fff92f34677 __NSThreadPerformPerform + 225
Find at the *lowest* call in MyFramework, line 5 in this case. As Nick Blievers pointed out to me, subtract the second number given from the first number to get the *Actual Load Address *A* of the framework.
A = 0x00000001000395db - 202203 = 0x100008000
4) Open a Terminal window and cd to this directory.
5) Run the following command:
otool -l MyApp.app/Contents/Frameworks/MyFramework.framework/MyFramework | grep -B 3 -A 8 -m 1 "__TEXT"
If that doesn't give 3+1+8=12 lines of text, that is because the -A and -B options in grep seem to be broken in Mountain Lion. Do the grep manually. Run this command:
otool -l MyApp.app/Contents/Frameworks/MyFramework.framework/MyFramework
then scroll up to the top of the result and read the first 12 lines.
You will get output which looks like this:
Load command 0
cmd LC_SEGMENT
cmdsize 464
segname __TEXT
vmaddr 0x00000000
vmsize 0x00111000
fileoff 0
filesize 1118208
maxprot 0x00000007
initprot 0x00000005
nsects 6
flags 0x0
7) Note the value given for vmaddr. We call this the *Intended Load Address* and denote it as *I*. I've never seen anything other than 0x0.
8) Calculate the *slide* = A - I
9) Enter "gdb" with no options, unless your crash report is from a different architecture. Then, you must make sure you use the correct -arch flag. Example, if the crash occurred on a PowerPC and you are not running now on a PowerPC: 'gdb -arch ppc'. If the crash occurred when running a 32-bit app on an Intel Mac and you are running on a 64-bit Intel Mac: 'gdb -arch i386'
10) Enter the command 'set sharedlibrary preload-libraries no'. This disables the behavior described above.
11) Load your binary with the 'file' command.
Example for main executable:
file MyApp.app/Contents/MacOS/MyApp
Example for framework (note that it's OK to use the symlink in the framework)
file MyApp.app/Contents/Frameworks/MyFramework.framework/MyFramework
12) Load the relevant dSYM library, for example:
add-dsym MyFramework.framework.dSYM
(Note: Usually this step is not necessary because gdb will find it in a Spotlight search under the hood. But Spotlight is only 80% reliable.
13) Look at the crash report and find the first interesting address in the call stack. For example, in
12 com.sheepsystems.MyApp 0x0002ef5f 0x1000 + 254908
the address is 0x0002ef5f
14) If you calculated a slide, tell gdb to get info for the address minus the slide:
(gdb) info line *(<address> - <slide>)
Example:
(gdb) info line *(0x0002ef5f - 0x1000)
If gdb has an answer, it will display it like this:
Line 117 of "/Users/jk/Documents/Programming/Projects/MyApp/MyAppTableColumn.m"
starts at address 0x2ef4e <-[MyAppTableColumn dataCellForRow:]+1033>
ends at 0x2ef61 <-[MyAppTableColumn dataCellForRow:]+1052>
Repeat the preceding two steps for other interesting addresses
For further information, read this thread:
http://www.cocoabuilder.com/archive/message/xcode/2008/2/26/19750
For further information, read this thread:
http://www.cocoabuilder.com/archive/message/xcode/2008/2/26/19750
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Xcode-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden