May 2011 - Darwin-kernel - lists.apple.com

Re: How to debug deadlocks in KEXT?
by Ken Hornstein 26 May '11

26 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >I am working on my own own KEXT that implements virtual file system >API. And I have a deadlock that I would like to understand and fix it. First, if you don't have it, you should get the Kernel Debug Kit corresponding to the kernel version you're using. For kernel versions up to 10.6.2, you can get them here: https://developer.apple.com/hardwaredrivers/download/kerneldebugkits.html For versions later than that ... well, I don't quite understand why, but Apple almost seems to be going out of their way to hide them. Use your ADC account and head over to http://connect.apple.com, and under "Downloads" select "MacOS X". At the very least, you're going to need the kgmacros included in that package. Others have already mentioned stackshot, but perhaps it's just me ... I never got it to produce anything useful (like symbolic names in my kext). I guess it's possible, so it was probably something I was doing wrong. However, I guess not everyone is aware of it but you CAN do limited local kext debugging via gdb. From the Kernel Debug Kit Read Me: [...] Live (single-machine) kernel debugging was introduced in Mac OS X Leopard. This allows limited introspection of the kernel on a currently-running system. This works using the normal kernel and the symbols in this Kernel Debug Kit by specifying kmem=1 in your boot-args; the DEBUG kernel is not required. To use the live kernel debugger, do the following: $ sudo nvram boot-args="<your_current_boot-args> kmem=1" $ sudo shutdown -r now After rebooting, do: $ sudo gdb --arch=<arch> --quiet /mach_kernel (gdb) target darwin-kernel (gdb) source /Volumes/KernelDebugKit/kgmacros Loading Kernel GDB Macros package. Type "help kgm" for more info. (gdb) attach [...] Now it turned out that in my case this wasn't that useful to me in practice; I ended up needing to do breakpoints and examing kernel panics so often that I never ended up trying it. Obviously you can't do breakpoints via this method, but you could (in theory) inspect a kernel thread to see where it is blocked. I think you'll find you probably need to do two-machine debugging at some point. >I guess this is because my kext (that runs on a target machine) is not >really in the debug mode. My kext was loaded fine, and the article >assumes that the kext has a kernel panic (that stops kernel for the >debugging). > >So my question is: How to debug a kernel extension when it does not >produce a kernel panic. What I want is to setup breakpoints in some >functions and see variables/parameters. Are there any additional >advises about debugging deadlocks? It's not mentioned very well, but you see the part in the document about setting the boot-args argument? The key there is to set the flag that triggers the machine to drop into the debugger when you want it to; the document which talks about that in a bit more detail is here: http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/Ker… On the machines I use, once you set the DB_NMI flag you can drop into the debugger by hitting the power button. Once you're in the debugger on the target machine, you can then use gdb to attach to it; at that point the kernel is stopped and you can set breakpoints, examine variables, etc etc. I have a script which scp's over my freshly-built kexts, loads them with kextutil -s, then scp's back the symbol files to the development machine so I can load them into gdb with add-kext. Very very quickly, I suspect what you want to do is this: - Set up your debugging environment (see above) - Find which process your thread is in via the kgmacro showalltasks If your thread doesn't have a user task associated with it, it will be in the kernel_task. - Figure out which thread is the blocked thread via showtaskthreads - Switch to that thread via switchtoact - At that point you'll be on the thread's stack and you can go from there. Hope this helps! --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: How to debug deadlocks in KEXT?
by Nick Blievers 26 May '11

26 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: How to debug deadlocks in KEXT?
by Quinn "The Eskimo!" 26 May '11

26 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 26 May 2011, at 21:17, Joel Reymont wrote: > You cannot debug a kext on a local machine. Agreed. However, if you're hunting a deadlock, and the deadlock doesn't affect the overall system too badly, <x-man-page://1/stackshot> can be very useful. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: How to debug deadlocks in KEXT?
by Joel Reymont 26 May '11

26 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=9QDUIT9eTDH8E4iq6lBlwFRk5tnx2yCG2Mh4XETToO4=; b=tKMlHXsajIYa8sFvuE4A2DL+DhMgrNyKUDvsa9TREE40e4GMC2davM5kh3UQW/rlyQ oIt11KARku8I/9pYTBRl8UU2Zr4v4nzg6SawPVfaxAiSMp9U5WeXS3S6ctTIhWNLM/k2 PeVk+0eZyVZbSshjYTk/duFETR+V2C1i97ZYA= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=tjUR77IIGwkGqwEvpZ23rvqn4OJWXA9cEU+QjZELI5XeXDJhsMrAwiICojZP1yDY76 FL6FIaXtW5PIGTPecpPYgQdW0s/5igq14NK8qslIrbLsEsbZJUMXVhMdTzZF+skJsN7S 144NartoWX2RllpFGWduKjXC3uSAi9GqoH6M4= On May 26, 2011, at 9:54 PM, Anatol Pomozov wrote: > What I want is to setup breakpoints in some > functions and see variables/parameters. Worked fine for me last time I tried. > Another question that is related to the article: how to debug KEXT on > a single machine? Is it possible to debug deadlocks locally? You cannot debug a kext on a local machine. Not unless you call debugging the adding of copious logging to the driver and analyzing of this output. -------------------------------------------------------------------------- - for hire: mac osx device driver ninja, kernel extensions and usb drivers ---------------------+------------+--------------------------------------- http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont ---------------------+------------+--------------------------------------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

How to debug deadlocks in KEXT?
by Anatol Pomozov 26 May '11

26 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=i/H8miIDusuu5pDPDh1d9nsWfk4Dk0gWUc8aXEkknIM=; b=HIA5NE3+n3uktss0lyuogdfYqI4jpaL0D9KA/eyd2HA1JHtlJL3nVPgjlADakqzuzg xAWuQkBcSj9DXS/ftO5yGycMDcKGJN1ZSXeHqZThNk7bEkvZOJURIZS4AJISMa5o28+w h+74GWEHwultNO+hEaTdLpX8F45EIQCcgPPB4= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=fpm+xInQ//bjH42+suB99WgJHQeUic8qKE/6SCIXzdyHAGL6IL5r7dJ/Y8wH/95Wbm 9XnQrYnrt+Ju3l6mr+IwyzHAJsB9uTZXZmPRbLtK6J24FHsHDmycF4XJ3NnuriPU5Hyq Xchl48wKYvygneBSdOL2Y61Jgi42kkNsB/D+8= Hi, I am working on my own own KEXT that implements virtual file system API. And I have a deadlock that I would like to understand and fix it. I googled about kext debugging and basically the only document about it is "Debugging a Kernel Extension with GDB" http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KEX… I went through the steps and on this step (gdb) kdp-reattach targetmachine gdb fails with following message: kdp_reply_wait: error from kdp_receive: receive timeout exceeded kdp_transaction (remote_connect): transaction timed out kdp_transaction (remote_connect): re-sending transaction I guess this is because my kext (that runs on a target machine) is not really in the debug mode. My kext was loaded fine, and the article assumes that the kext has a kernel panic (that stops kernel for the debugging). So my question is: How to debug a kernel extension when it does not produce a kernel panic. What I want is to setup breakpoints in some functions and see variables/parameters. Are there any additional advises about debugging deadlocks? Another question that is related to the article: how to debug KEXT on a single machine? Is it possible to debug deadlocks locally? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Determine a process that "owns" the socket_t
by James J 23 May '11

23 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=2zT2GW/T9WYIl7NGuvMdQRZeqFYlm6u8eATKKHfQxNc=; b=HChBh3ESu0Ynqj5cY41ZS47TaKzjnf5sJ0FaYfRb/9AlR/rcz28JN3LW3oD/GSdFz+ zRSm4Expj4BSTfUQxFj6K4hpy0Gw70MzS3McBL4wwCGOPxTWcjuITOa/tOpEah3shdhG yhuhetBmq8UgElp9EjHJaTGNdIshWEk+EHMdo= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=NpXWglDSR75VnuXyVooMdy2Y7VCEQD1b/tPBDDCyBcgDCWec0GMRK+s0Fc+hTKHx1v Q8M7kEDiJcgjpJggTmt84oLi1tMhGYA++IyQmp+k+vPDySM9wQwEXrhFDciY4df/QKX+ 7CRyE5PGDS7PRT/qKMvNzejCVH/foxOvfMDv4= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Determine a process that "owns" the socket_t
by Ken Hornstein 23 May '11

23 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >The closest thing i came up with - is to somehow "ask" the process if it is >"mine", right in "filter_attached_to_socket" NKE's callback. > I am searching for the way to 'ask' it, it should be some kind of IPC, but >i am not sure which one. The only thing i was using for kernel-userland >communication were Control Sockets, but the conversation for these is >initiated by an application not the kernel extension (while i need a driver >to ask the application, not vice versa, - of course with all error checking >and "app not responding/not loaded" correct processing). I am not even sure >if that is possible to do at all. The way I've seen this done (using other mechanisms) is to have your userspace code have a thread that makes a call into the kernel and the kernel call just blocks. When the kernel wants to ask the userspace code to do something, it does that by having the blocked userspace call complete. Seems like that could work here. The only other option ... well, I'm not sure you can easily do it yourself, but look at bsd/nfs/nfs_gss.c, specifically nfs_gss_clnt_gssd_upcall(). I would caution you that I'm not sure that you can make use of that yourself, but it's at least worth looking at. --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Determine a process that "owns" the socket_t
by James J 22 May '11

22 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Lf1KykWQ1JNciz0IO+lkZdA3z1+w2GndHQXu/ODEOc8=; b=SdTC+V96Wa0idAFTz21q1FEGTc1ycC8AHWm/E6ZOVP30mjtm3UsDOUHLTbNR+dXeub +l33QlSBizWdBwGtppRSE0Tg14m2nLyoNvhN1mMCnv2ic06gAIm3C/hEY0pUP41Ibviz GKZQfpffnhpqG/F3PSupPCDrlYJ7iyYAN/HGU= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=t7n39LVqszfA8OVvfG/ZOfzPKNJiO4EmLI4JuN5Jm0zDTzzCQgCQspybWSynNuU+XD b/KtATKZYRb4hiCqCxq37GCxMM8vdn4BUYGP5vWk0sXxgWKj+0T9oddKTil+IsWm4T0D /rneUqH3/xXXABlO2qsKYjeqoOJ9qFUa90ZnI= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

KAUTH_FILEOP_DELETE, KAUTH_FILEOP_LINK, KAUTH_FILEOP_SYMLINK
by Christopher James Elphinstone Chandler 20 May '11

20 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: The correct way to identify applications in a kext
by Quinn "The Eskimo!" 12 May '11

12 May '11

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 12 May 2011, at 14:22, Juuso Salonen wrote: > This user-space-helper approach is something I feel queasy about, > although it is probably caused by my lack of understanding. Simply > put, isn't it way too slow? Not particularly. As I mentioned in my email, you only do this once per process, on the assumption that a process's code signature won't change on the fly [1]. One round trip to user space the first time the process makes an outgoing connection is not going to impact performance meaningfully. The built-in firewall uses essentially the same technique, and we've not had a lot of complaints (about the performance, anyway ;-). Oh, some extra points: o Every time you write kernel code that depends on user space code, you have to be very careful about deadlocks. o You can avoid a world of potential deadlocks by, as a matter of policy, /not/ checking the signature for any code running as root. This prevents you from deadlocking against critical system processes (the vnode pager, the DirectoryService daemon, and so on). o One final consequence of this design is that you must decide on a reasonable default policy if the user space code has not checked in (that is, before the launchd daemon has run) or has failed or just does not reply to your requests in a timely fashion. What that default policy should be is up to you. Common approaches include a) unilaterally denying the request, b) unilaterally allowing the request, and c) kernel panicking (for those in high-security environments). S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware [1] Which isn't a totally valid assumption, but it's valid enough for your purposes. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0