March 2007 - Darwin-kernel - lists.apple.com

Finding the device in question
by Nils Holland 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi everybody, Greetings, Nils _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… well, excuses in advance if this question is too simple and / or off- topic, but actually, I'd appreciate some hints on the following: Let's assume I have the following line in my system.log file on an Intel-Mac running MacOS X 10.4.8: | Mar 7 08:59:21 Nils-Hollands-MacBook-Pro kernel[0]: [CompletePowerStateChange] Failed to resetDevice 00000010 In that case, how do I actually find out which device is being referred to here (00000010)? Background: I've been trying to find out about various sleep-related (or rather "randomly waking from sleep") relates issues here lately. The details about these certainly don't belong onto this mailing list, so I won't post them here. Still, the above mentioned line from my system.log is a trail I'm currently following, so any hints on where to look in order to find out what's going on here would be much appreciated! This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Rob McKeever 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -Rob -- Rob McKeever, rmckeever(a)apple.com Developer Technical Support, CoreOS/Hardware Apple Computer On Mar 7, 2007, at 9:54 AM, Bhavesh Davda wrote: Sorry for not providing more context. Yes, what I'm looking for is something that has memory semantics (i.e. I don't need to call map/unmap functions ever time I want to access this piece of memory), where the memory is allocated in a user space task, but should be accessible without these map/unmap calls like "global" memory within a KEXT, for all tasks running the KEXT code as part of a synchronous system call, and also for all kernel threads running the KEXT code as some sort of a "bottom-half". I understand that in the context of the userspace task that allocates and owns the memory, the kernel (KEXT) doesn't need any special mapping code to access the tasks userspace memory. I'm also not looking for the bounce buffer element of Linux's kmap. All I would like is a way to, through non-kernel-private APIs, be able to map the userspace memory into the kernel map, so that all tasks can access it in KEXT code. Hopefully this makes the intent more clear. Thank you! - Bhavesh On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: > If you are not working with a file system then I would suggest > checking out what ubc_upl_map() does and following it up that way > to figure out what to do analogously. That's probably not a good suggestion; the UBC implementation does not use KPI. If the page is sourced from an address space (as I inferred from the Bhavesh's original request) then there's no need to construct the UPL (it will happen automagically if the page has to be faulted in). What this boils down to, again, is that without more information on what Bhavesh actually wants to do (i.e. not "there is this call in Linux"), we're not really able to help him very much. 8) = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This sounds like a perfect use of an I/O Kit IOMemoryDescriptor. Have you looked at this and would it be an option for you? On 3/7/07, Michael Smith <drivers(a)mu.org> wrote: -- Bhavesh P. Davda _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/rmckeever% 40apple.com This email sent to rmckeever(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Bhavesh Davda 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=PNKvP6xK3+///EF7fHdusNSmXwGzpI/R0cdq3YDS/Y5iRHptT43Myeteg2K9cNJno5gjY6c5czT58hAd+CkTOGiHoImpcecS0xiXtKKniMjKvro47/AUubZmDml1V1PXT7ySGE/8SzWt4N9ywTbVo3X/InMXze7N/y2lNmkMi/U= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=CNOpjRwEmozyahnKsxXbZ+V/nkTSTvbfyHAxXzrs4WGIaRsq6Aba69v012tUS+beiY0+WGXyc2f0L7DuC5uARREFugyo0CVuFD/w99r5R6ttktJ4Xwsh4mj4PlWRt6MX8Ra1NwNQ2wWMlZhzmSdPOT6kUyoqfaeODo74ntj8tb8= Sorry for not providing more context. Yes, what I'm looking for is something that has memory semantics (i.e. I don't need to call map/unmap functions ever time I want to access this piece of memory), where the memory is allocated in a user space task, but should be accessible without these map/unmap calls like "global" memory within a KEXT, for all tasks running the KEXT code as part of a synchronous system call, and also for all kernel threads running the KEXT code as some sort of a "bottom-half". I understand that in the context of the userspace task that allocates and owns the memory, the kernel (KEXT) doesn't need any special mapping code to access the tasks userspace memory. I'm also not looking for the bounce buffer element of Linux's kmap. All I would like is a way to, through non-kernel-private APIs, be able to map the userspace memory into the kernel map, so that all tasks can access it in KEXT code. Hopefully this makes the intent more clear. Thank you! - Bhavesh On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: > If you are not working with a file system then I would suggest > checking out what ubc_upl_map() does and following it up that way > to figure out what to do analogously. That's probably not a good suggestion; the UBC implementation does not use KPI. If the page is sourced from an address space (as I inferred from the Bhavesh's original request) then there's no need to construct the UPL (it will happen automagically if the page has to be faulted in). What this boils down to, again, is that without more information on what Bhavesh actually wants to do (i.e. not "there is this call in Linux"), we're not really able to help him very much. 8) = Mike -- Bhavesh P. Davda _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 3/7/07, Michael Smith <drivers(a)mu.org> wrote: This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Anton Altaparmakov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: [ ... ] (int)kerr); err = EIO; I may be misunderstanding the simplification you performed, and if so, forgive me... By the way, is there any openly available documentation on UPL interface and XNU VM internals in general? - Really excellent book!: MAC OS X Internals by Amit Singh (http:// www.amazon.co.uk/MAC-OS-Internals-Amit-Singh/dp/0321278542/) You are welcome. - Download the xnu sources from http://www.opensource.apple.com/ darwinsource/ and look in bsd/kern/ubc_subr.c. It has very good comments above most of the function (kind of Kernel doc style like in Linux but not quite). Oops. The nice comments are not in Tiger. )-: Sorry. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Mar 2007, at 09:20, Nikita Danilov wrote: 07/03/2007 13:36 +0000, Anton Altaparmakov: On 7 Mar 2007, at 07:52, Nikita Danilov wrote: 07/03/2007 04:37 -0800, Terry Lambert: [...] You want to call ubc_upl_map() before you do the I/O, or the UPL could have been recycled while the system was under extreme pressure, which - http://developer.apple.com/referencelibrary/Darwin/idxKernel- date.html and in particular the Kernel Programming Guide (http:// developer.apple.com/documentation/Darwin/Conceptual/ KernelProgramming/). Thanks, Anton, I'll take a look. A couple of years ago, I used to read through osfmk/vm/ a lot, trying to understand UPL interface, hopefully it got better comments (read: any) since then. :-) I should note that I am looking at xnu trunk in Apple internal svn, i.e. what will be in Leopard when it is released. /me checks out the current Tiger 10.4.8 xnu tag (xnu-792.14.14) and looks at bsd/kern/ubc_subr.c(). I wonder if it would be allowed to cut the comments out of the current ubc_subr.c and make them public as a .txt file... /me writes a Sticky to remind me to ask... This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Anton Altaparmakov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: [ ... ] Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Mar 2007, at 12:37, Terry Lambert wrote: /* Create a page list for the wanted page. */ kerr = ubc_create_upl(ni->vn, ofs, PAGE_SIZE, upl, pl, UPL_SET_LITE | (rw ? UPL_WILL_MODIFY : 0)); if (kerr != KERN_SUCCESS) panic("%s(): Failed to get page (error %d).\n", __FUNCTION__, (int)kerr); /* * If the page is not valid, need to read it in from the vnode now thus * making it valid. */ if (!upl_valid_page(*pl, 0)) { ntfs_debug("Reading page as it was not valid."); err = ntfs_pagein(ni, ofs, PAGE_SIZE, *upl, 0, UPL_IOSYNC | UPL_NOCOMMIT); if (err) { ntfs_error(ni->vol->mp, "Failed to read page (error " "%d).", err); goto pagein_err; } } /* Map the page into the kernel's address space. */ kerr = ubc_upl_map(*upl, (vm_offset_t*)kaddr); if (kerr == KERN_SUCCESS) { ntfs_debug("Done."); return 0; } ntfs_error(ni->vol->mp, "Failed to map page (error %d).", (int)kerr); err = EIO; I may be misunderstanding the simplification you performed, and if so, forgive me... You want to call ubc_upl_map() before you do the I/O, or the UPL could have been recycled while the system was under extreme pressure, which would result in it being thrown away before a mapping was established to hold it in place. It looks like the failure mode in this case would cause a ""Failed to map page" message if debugging was on, and an EIO, which I suspect would not be re-driven down. Um, surely once the ubc_create_upl() call succeeds I have total and exclusive control of the UPL and _nothing_ can happen to it at all unless I do it myself for as long as I have not released the UPL back to the VM. Any processes that try to do anything with the pages in my UPL will block because UPLs provide exclusive access to the pages until I either abort or commit the UPL. The ubc_upl_map() only provides a mapping of the pages into a contiguous virtual memory "segment" (or maybe better called "region" to avoid confusion with a CPU segment...). It does not do anything else. If what you say were to be correct the cluster i/o layer would not actually work as that works with the UPLs without mapping the pages for the majority of the time (ubc_upl_map() is very heavy weight and has to be avoided if/where possible). For example see bsd/vfs/ vfs_cluster.c::cluster_zero() which operates directly on the underlying physical memory without ever mapping it into kernel virtual address space as well as bsd/vfs/ vfs_cluster.c::cluster_copy_upl_data() which does uiomove64() directly on the physical memory. Even if the EIO caused to to be redriven down instead of returning an error up the stack, potentially to the program on the other end of a system call, you might still not be very happy, if the system stays under pressure, since it can happen repeatedly, if the load is sustained. You probably want to look to bsd/vfs/vfs_bio.c and how HFS does things for your example (e.g. the usage of ubc_upl_map () in buf_map() & company). You probably also want to drive the I/O through the fault on access, through the page-in path, rather than an explicit I/O through ntfs_pagein(). The reason the cd9660 code does it the other way is because CD's don't have normal page-aligned block boundaries, so avoiding the direct I/O and letting the VM system schedule instead could be inefficient. I do not think that is possible / would work at all as no page fault would be triggered once you have created the UPL. The page is yours and yours alone at that point nothing including the kernel itself can touch it or do anything at all to it. This email sent to site_archiver(a)lists.apple.com

1 0

Re: KEXT/KAUTH, malloc-related crash, login only?!?
by Ondra Cada 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry, On 7.3.2007, at 13:40, Terry Lambert wrote: alloc_item if (OK) { alloc_and_fill_names if (OK) { if (limit_reached) dequeue_last enqueue_item } } _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… It might be important that, like Kauthorama does, when the limit is reached I don't cease malloc'ing; instead, for each malloc I release the oldest queued item. Essentially, the algorithm is How do you protect your queue against reentrancy by multiple threads of control, resulting in corruption of the queue? Using locks (lck_mtx_lock/unlock). That's irrelevant to the problem though: the KEXT crashes even if the queue access is completely commented out. --- Ondra Čada OCSoftware: ocs(a)ocs.cz http://www.ocs.cz private ondra(a)ocs.cz http://www.ocs.cz/oc This email sent to site_archiver(a)lists.apple.com

1 0

Re: Cross Debugging
by Andrew Gallatin 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Michael Smith writes: > > On Mar 5, 2007, at 2:56 AM, Sven Mutke wrote: > > > My question: is there a recommended way to set up Linux or FreeBSD to > > debug the kernel on the OSX machine? > > No. Unless your goal is to get hopelessly frustrated debugging gdb, > go buy an older Mac and use it to crash-test your kernel changes. > Well, it wasn't *that* bad. I built the darwin gdb on FreeBSD 4.x years ago, and I remember it being less than an afternoon's worth of work. I've still got the binary (http://people.freebsd.org/~gallatin/macgdb.gz) but I doubt it would work against tiger. However, if I was doing it today, I'd just build kdumpd on the linux or FreeBSD host and use remote kernel crash dumps (http://developer.apple.com/technotes/tn2004/tn2118.html) Building it is trivial, as kdumpd seems to have been ported from tftpd. That way, you can debug on your mac, after it has rebooted. And there is always the possibility to run darwin inside a virtual machine.. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Anton Altaparmakov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: [ ... ] (int)kerr); err = EIO; Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Mar 2007, at 07:52, Nikita Danilov wrote: 07/03/2007 04:37 -0800, Terry Lambert: [...] I may be misunderstanding the simplification you performed, and if so, forgive me... You want to call ubc_upl_map() before you do the I/O, or the UPL could have been recycled while the system was under extreme pressure, which By the way, is there any openly available documentation on UPL interface and XNU VM internals in general? - Really excellent book!: MAC OS X Internals by Amit Singh (http:// www.amazon.co.uk/MAC-OS-Internals-Amit-Singh/dp/0321278542/) - http://developer.apple.com/referencelibrary/Darwin/idxKernel- date.html and in particular the Kernel Programming Guide (http:// developer.apple.com/documentation/Darwin/Conceptual/KernelProgramming/) - Download the xnu sources from http://www.opensource.apple.com/ darwinsource/ and look in bsd/kern/ubc_subr.c. It has very good comments above most of the function (kind of Kernel doc style like in Linux but not quite). This email sent to site_archiver(a)lists.apple.com

1 0

Re: KEXT/KAUTH, malloc-related crash, login only?!?
by Terry Lambert 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Tried different values, the last one 200. alloc_item if (OK) { alloc_and_fill_names if (OK) { if (limit_reached) dequeue_last enqueue_item } } -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Mar 5, 2007, at 12:30 PM, Ondra Cada wrote: struct file_change { TAILQ_ENTRY(file_change) link; char *name,*oldname; // malloc'd, to be released, oldname only for RENAME unsigned long long serial; // ID of this item long pid; // caller process ID }; In "full" version of course there's much more alloc'd for the names. It, though, crashes the same when the name allocation is commented- out. How large was the limit you imposed? It might be important that, like Kauthorama does, when the limit is reached I don't cease malloc'ing; instead, for each malloc I release the oldest queued item. Essentially, the algorithm is How do you protect your queue against reentrancy by multiple threads of control, resulting in corruption of the queue? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Terry Lambert 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: [ ... ] -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… /* Create a page list for the wanted page. */ kerr = ubc_create_upl(ni->vn, ofs, PAGE_SIZE, upl, pl, UPL_SET_LITE | (rw ? UPL_WILL_MODIFY : 0)); if (kerr != KERN_SUCCESS) panic("%s(): Failed to get page (error %d).\n", __FUNCTION__, (int)kerr); /* * If the page is not valid, need to read it in from the vnode now thus * making it valid. */ if (!upl_valid_page(*pl, 0)) { ntfs_debug("Reading page as it was not valid."); err = ntfs_pagein(ni, ofs, PAGE_SIZE, *upl, 0, UPL_IOSYNC | UPL_NOCOMMIT); if (err) { ntfs_error(ni->vol->mp, "Failed to read page (error " "%d).", err); goto pagein_err; } } /* Map the page into the kernel's address space. */ kerr = ubc_upl_map(*upl, (vm_offset_t*)kaddr); if (kerr == KERN_SUCCESS) { ntfs_debug("Done."); return 0; } ntfs_error(ni->vol->mp, "Failed to map page (error %d).", (int)kerr); err = EIO; I may be misunderstanding the simplification you performed, and if so, forgive me... You want to call ubc_upl_map() before you do the I/O, or the UPL could have been recycled while the system was under extreme pressure, which would result in it being thrown away before a mapping was established to hold it in place. It looks like the failure mode in this case would cause a ""Failed to map page" message if debugging was on, and an EIO, which I suspect would not be re-driven down. Even if the EIO caused to to be redriven down instead of returning an error up the stack, potentially to the program on the other end of a system call, you might still not be very happy, if the system stays under pressure, since it can happen repeatedly, if the load is sustained. You probably want to look to bsd/vfs/vfs_bio.c and how HFS does things for your example (e.g. the usage of ubc_upl_map() in buf_map() & company). You probably also want to drive the I/O through the fault on access, through the page-in path, rather than an explicit I/O through ntfs_pagein(). The reason the cd9660 code does it the other way is because CD's don't have normal page-aligned block boundaries, so avoiding the direct I/O and letting the VM system schedule instead could be inefficient. This email sent to site_archiver(a)lists.apple.com

1 0