Re: How to debug deadlocks in KEXT?
Re: How to debug deadlocks in KEXT?
- Subject: Re: How to debug deadlocks in KEXT?
- From: Ken Hornstein <email@hidden>
- Date: Thu, 26 May 2011 19:19:35 -0400
>I am working on my own own KEXT that implements virtual file system
>API. And I have a deadlock that I would like to understand and fix it.
First, if you don't have it, you should get the Kernel Debug Kit
corresponding to the kernel version you're using. For kernel
versions up to 10.6.2, you can get them here:
https://developer.apple.com/hardwaredrivers/download/kerneldebugkits.html
For versions later than that ... well, I don't quite understand
why, but Apple almost seems to be going out of their way to hide
them. Use your ADC account and head over to http://connect.apple.com,
and under "Downloads" select "MacOS X". At the very least, you're going
to need the kgmacros included in that package.
Others have already mentioned stackshot, but perhaps it's just me ...
I never got it to produce anything useful (like symbolic names in
my kext). I guess it's possible, so it was probably something I was
doing wrong.
However, I guess not everyone is aware of it but you CAN do limited
local kext debugging via gdb. From the Kernel Debug Kit Read Me:
[...]
Live (single-machine) kernel debugging was introduced in Mac OS X
Leopard. This allows limited introspection of the kernel on a
currently-running system. This works using the normal kernel and
the symbols in this Kernel Debug Kit by specifying kmem=1 in your
boot-args; the DEBUG kernel is not required.
To use the live kernel debugger, do the following:
$ sudo nvram boot-args="<your_current_boot-args> kmem=1"
$ sudo shutdown -r now
After rebooting, do:
$ sudo gdb --arch=<arch> --quiet /mach_kernel
(gdb) target darwin-kernel
(gdb) source /Volumes/KernelDebugKit/kgmacros
Loading Kernel GDB Macros package. Type "help kgm" for more info.
(gdb) attach
[...]
Now it turned out that in my case this wasn't that useful to me in
practice; I ended up needing to do breakpoints and examing kernel
panics so often that I never ended up trying it. Obviously you
can't do breakpoints via this method, but you could (in theory)
inspect a kernel thread to see where it is blocked. I think you'll
find you probably need to do two-machine debugging at some point.
>I guess this is because my kext (that runs on a target machine) is not
>really in the debug mode. My kext was loaded fine, and the article
>assumes that the kext has a kernel panic (that stops kernel for the
>debugging).
>
>So my question is: How to debug a kernel extension when it does not
>produce a kernel panic. What I want is to setup breakpoints in some
>functions and see variables/parameters. Are there any additional
>advises about debugging deadlocks?
It's not mentioned very well, but you see the part in the document
about setting the boot-args argument? The key there is to set the
flag that triggers the machine to drop into the debugger when you
want it to; the document which talks about that in a bit more detail
is here:
http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221
On the machines I use, once you set the DB_NMI flag you can drop into the
debugger by hitting the power button.
Once you're in the debugger on the target machine, you can then use gdb
to attach to it; at that point the kernel is stopped and you can set
breakpoints, examine variables, etc etc.
I have a script which scp's over my freshly-built kexts, loads them with
kextutil -s, then scp's back the symbol files to the development machine
so I can load them into gdb with add-kext.
Very very quickly, I suspect what you want to do is this:
- Set up your debugging environment (see above)
- Find which process your thread is in via the kgmacro showalltasks
If your thread doesn't have a user task associated with it, it will
be in the kernel_task.
- Figure out which thread is the blocked thread via showtaskthreads
- Switch to that thread via switchtoact
- At that point you'll be on the thread's stack and you can go from there.
Hope this helps!
--Ken
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden