• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: interpreting kernel panic logs on Intel
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: interpreting kernel panic logs on Intel

  • Subject: Re: interpreting kernel panic logs on Intel
  • From: Garth Cummings <email@hidden>
  • Date: Fri, 1 Dec 2006 14:12:16 -0800
Hi Stuart,

On Dec 1, 2006, at 1:07 PM, Stuart Smith wrote:

Apple have a nice Technical Note here:

http://developer.apple.com/technotes/tn2002/tn2063.html

which teaches one how to interpret the backtraces from kernel panic logs on
PowerPC, but no such document for Intel.

Yeah, it's on the list to do.

The process is generally the same on Intel-based Macs as on their PowerPC-based counterparts. The main difference is that while instructions on PowerPC are always four bytes long, on Intel processors the instruction length varies. So the info in the technote about subtracting 4 from the return addresses in the backtrace won't work. Instead, disassemble the function in question and look at the instruction prior to the return address to see the call site. 

I have a panic log from an intel machine. I've made symbol files for the
"Kernel loadable modules in backtrace", but the backtrace is in the format:

Format - Frame : Return Address (4 potential args on stack)

e.g.

0x485ebc08 : 0x128d1f (0x3c9540 0x485ebc2c 0x131df4 0x0)

what do I plug into gdb to get the symbol name at that address? Everything
I've tried so far has resulted in this type of message:

(gdb) x/i 0x485ebc08
0x485ebc08:     Cannot access memory at address 0x485ebc08
(gdb) x/i 0x128d1f
0x128d1f:       Cannot access memory at address 0x128d1f

That "Cannot access memory at address" error occurs if the address is one that GDB doesn't know about. Given how low in memory that address is, it's almost certainly in the kernel. You can get those kernel symbols required for external linkage just by "add-symbol-file /mach_kernel". If you need more symbols, download the Kernel Debug Kit for the OS build you're running and use the mach_kernel from the KDK.

The backtrace header is a helpful reminder--translated it says "The format of each line is the frame pointer : return address followed by four potential arguments on the stack." So in your example your second try is nearly correct except that the symbol for that address isn't known to GDB. Also, on PowerPC that would have been "x/i 0x128d1f-4" (except that PowerPC instructions are always on even addresses). Since you can't tell what the previous Intel instruction is, do "disass 0x128d1f" instead and look at the instruction right before 0x128d1f.

Here's an example panic log:

panic(cpu 0 caller 0x001A3135): Unresolved kernel trap (CPU 0, Type 14=page fault), registers:
CR0: 0x8001003b, CR2: 0x00000000, CR3: 0x017c7000, CR4: 0x000006e0
EAX: 0x046a7400, EBX: 0x03104ac0, ECX: 0x03a498b0, EDX: 0x00000000
CR2: 0x00000000, EBP: 0x1483bce8, ESI: 0x046a7400, EDI: 0x00000000
EFL: 0x00010206, EIP: 0x0038478c, CS:  0x00000008, DS:  0x14830010

Backtrace, Format - Frame : Return Address (4 potential args on stack)
0x1483bb28 : 0x128d1f (0x3c9540 0x1483bb4c 0x131df4 0x0)
0x1483bb68 : 0x1a3135 (0x3cf1f4 0x0 0xe 0x3cea24)
0x1483bc78 : 0x19a8d4 (0x1483bc88 0x30004 0xe 0x48)
0x1483bce8 : 0x382345 (0x46a7400 0x3104ac0 0x356a7400 0x30303030)
0x1483bd08 : 0x24ab3806 (0x46a7400 0x24adb4a8 0x24 0x0)
0x1483bd58 : 0x2df576dc (0x2fa1c00 0x36c9f20 0x46a7400 0x50000)
0x1483bda8 : 0x2df567b8 (0x32c6000 0x2df5aa18 0x2f44cd0 0x12d9b3)
0x1483bdc8 : 0x24a44166 (0x32c6000 0x3393600 0x36b5d40 0x1)
0x1483bdf8 : 0x24a46cac (0x32c6000 0x3393600 0x1483be28 0x38cc9f)
0x1483be18 : 0x2df566bf (0x32c6000 0x3393600 0x1483be58 0x3865f3)
0x1483be58 : 0x38a656 (0x32c6000 0x3393600 0x1 0x3bdf0f)
0x1483beb8 : 0x38bed5 (0x3393600 0x32c6000 0x46c4fc0 0x3393600)
0x1483bf38 : 0x38b73d (0x3393600 0x367adc0 0x0 0x3c0dcb)
0x1483bf88 : 0x38ce82 (0x3393600 0x8 0x134d5e 0x38cda8)
0x1483bfc8 : 0x19a74c (0x37a6420 0x0 0x19d0b5 0x2f43c64) Backtrace terminated-invalid frame pointer 0x0
      Kernel loadable modules in backtrace (with dependencies):
         com.apple.iokit.IOPCCardFamily(1.6.2)@0x2df46000
            dependency: com.apple.iokit.IOPCIFamily(2.1)@0x24a42000
         com.apple.driver.AppleACPIPlatform(1.0.5)@0x24aab000
            dependency: com.apple.iokit.IOPCIFamily(2.1)@0x24a42000
            dependency: com.apple.iokit.IOACPIFamily(1.2.0)@0x24a58000
         com.apple.iokit.IOPCIFamily(2.1)@0x24a42000

Kernel version:
Darwin Kernel Version 8.8.1: Mon Sep 25 19:42:00 PDT 2006; root:xnu-792.13.8.obj~1/RELEASE_I386

To interpret this, we first need the symbol files for all of the dependencies. While IOPCCardFamily and IOPCIFamily are in the Kernel Debug Kit, the ACPI family and driver are not. So we can use the KDK for the first two KEXTs and the ACPI KEXTs from a running 10.4.8 system (the kernel version 8.8.1 translates to 10.4.8).

$ /Volumes/KernelDebugKit/createsymbolfiles -s /tmp/ /Volumes/KernelDebugKit/IOPCCardFamily.kext/
enter the hexadecimal load addresses for these modules:
com.apple.iokit.IOPCIFamily: 0x24a42000
com.apple.iokit.IOPCCardFamily: 0x2df46000
$ /sbin/kextload -c -e -k /Volumes/KernelDebugKit/mach_kernel -n -z -s /tmp/ /System/Library/Extensions/AppleACPIPlatform.kext
enter the hexadecimal load addresses for these modules:
com.apple.iokit.IOPCIFamily: 0x24a42000
com.apple.iokit.IOACPIFamily: 0x24a58000
com.apple.driver.AppleACPIPlatform: 0x24aab000

Now that the symbol files have been created in /tmp, we can start GDB and use it to decode the backtrace:

$ gdb /Volumes/KernelDebugKit/mach_kernel
...
(gdb) set print asm-demangle on
(gdb) add-symbol-file /tmp/com.apple.driver.AppleACPIPlatform.sym                              
add symbol table from file "/tmp/com.apple.driver.AppleACPIPlatform.sym"? (y or n) y
Reading symbols from /private/tmp/com.apple.driver.AppleACPIPlatform.sym...done.
(gdb) add-symbol-file /tmp/com.apple.iokit.IOACPIFamily.sym      
add symbol table from file "/tmp/com.apple.iokit.IOACPIFamily.sym"? (y or n) y
Reading symbols from /private/tmp/com.apple.iokit.IOACPIFamily.sym...done.
(gdb) add-symbol-file /tmp/com.apple.iokit.IOPCIFamily.sym
add symbol table from file "/tmp/com.apple.iokit.IOPCIFamily.sym"? (y or n) y
Reading symbols from /private/tmp/com.apple.iokit.IOPCIFamily.sym...done.
(gdb) add-symbol-file /tmp/com.apple.iokit.IOPCCardFamily.sym
add symbol table from file "/tmp/com.apple.iokit.IOPCCardFamily.sym"? (y or n) y
Reading symbols from /private/tmp/com.apple.iokit.IOPCCardFamily.sym...done.
(gdb) disass 0x128d1f
Dump of assembler code for function panic:
0x00128ba1 <panic+0>:   push   ëp
0x00128ba2 <panic+1>:   mov    %esp,ëp
...
0x00128d13 <panic+370>: movl   $0x3c9540,(%esp)
0x00128d1a <panic+377>: call   0x1a840b <Debugger>
0x00128d1f <panic+382>: mov    4317808,ìx
0x00128d25 <panic+388>: test   ìx,ìx
...
End of assembler dump.
(gdb) disass 0x1a3135
Dump of assembler code for function kernel_trap:
0x001a2b33 <kernel_trap+0>:     push   ëp
0x001a2b34 <kernel_trap+1>:     mov    %esp,ëp
...
0x001a3129 <kernel_trap+1526>:  movl   $0x3cf1f4,(%esp)
0x001a3130 <kernel_trap+1533>:  call   0x128ba1 <panic>
0x001a3135 <kernel_trap+1538>:  mov    4(êx),êx
0x001a3138 <kernel_trap+1541>:  mov    êx,56(íi)
...
End of assembler dump.
(gdb) disass 0x19a8d4
Dump of assembler code for function trap_from_kernel:
0x0019a8c1 <trap_from_kernel+0>:        mov    %esp,êx
0x0019a8c3 <trap_from_kernel+2>:        mov    %esp,íi
...
0x0019a8cb <trap_from_kernel+10>:       mov    êx,0(%esp)
0x0019a8cf <trap_from_kernel+14>:       call   0x1a2b33 <kernel_trap>
0x0019a8d4 <trap_from_kernel+19>:       mov    íi,%esp
0x0019a8d6 <trap_from_kernel+21>:       cli   
...
End of assembler dump.
(gdb) disass 0x382345
Dump of assembler code for function _ZNK15IORegistryEntry11getPropertyEPKc:
0x0038231e <IORegistryEntry::getProperty(char const*) const+0>: push   ëp
0x0038231f <IORegistryEntry::getProperty(char const*) const+1>: mov    %esp,ëp
...
0x0038233f <IORegistryEntry::getProperty(char const*) const+33>:        call   *324(êx)
0x00382345 <IORegistryEntry::getProperty(char const*) const+39>:        mov    êx,%esi
0x00382347 <IORegistryEntry::getProperty(char const*) const+41>:        mov    (ëx),êx
0x00382349 <IORegistryEntry::getProperty(char const*) const+43>:        mov    ëx,(%esp)
...
End of assembler dump.

and so forth.


tia, Stuart

Hope this helps,
--gc
____________________________________________________________________
Garth Cummings  email@hidden
Sr. Software Engineer
Apple Developer Technical Support

<http://developer.apple.com/technicalsupport>

 

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden
References: 
 >interpreting kernel panic logs on Intel (From: Stuart Smith <email@hidden>)

  • Prev by Date: Re: 64-bit aware vmmap?
  • Next by Date: Re: interpreting kernel panic logs on Intel
  • Previous by thread: interpreting kernel panic logs on Intel
  • Next by thread: Re: interpreting kernel panic logs on Intel
  • Index(es):
    • Date
    • Thread