March 2007 - Darwin-kernel - lists.apple.com

Re: Equivalent of Linux kmap on Mac OS
by Anton Altaparmakov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: That is very true indeed! (-: Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Mar 2007, at 12:25, Michael Smith wrote: If you are not working with a file system then I would suggest checking out what ubc_upl_map() does and following it up that way to figure out what to do analogously. That's probably not a good suggestion; the UBC implementation does not use KPI. If the page is sourced from an address space (as I inferred from the Bhavesh's original request) then there's no need to construct the UPL (it will happen automagically if the page has to be faulted in). What this boils down to, again, is that without more information on what Bhavesh actually wants to do (i.e. not "there is this call in Linux"), we're not really able to help him very much. 8) This email sent to site_archiver(a)lists.apple.com

1 0

Re: KEXT/KAUTH, malloc-related crash, login only?!?
by Michael Smith 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 5, 2007, at 12:30 PM, Ondra Cada wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Have you set a debug= boot-arg so that you can get the actual crash/panic message? This would be helpful. Nope, haven't know of this. Thanks, will do! This will need to be the next step; everything else you've described sounds sane. You might want to double-check that you're not running off the end of your allocated structure, but at 24 bytes you should be getting entries from the 32-byte zone which has room for far more than 200 items so we need to see how it's actually crashing. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Michael Smith 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… If you are not working with a file system then I would suggest checking out what ubc_upl_map() does and following it up that way to figure out what to do analogously. That's probably not a good suggestion; the UBC implementation does not use KPI. If the page is sourced from an address space (as I inferred from the Bhavesh's original request) then there's no need to construct the UPL (it will happen automagically if the page has to be faulted in). What this boils down to, again, is that without more information on what Bhavesh actually wants to do (i.e. not "there is this call in Linux"), we're not really able to help him very much. 8) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Terry Lambert 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com What's the equivalent of Linux's kmap to map a user virtual address into the kernel memory space so that it is accessible "magically" (i.e. without any additional mapping/unmapping required) to all other tasks and kernel threads? man fcntl <http://lwn.net/Articles/111226/> -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Mar 6, 2007, at 4:35 PM, Bhavesh Davda wrote: I've already looked at vm_map and AFAICT, that ain't it. The concept of kmap() is specifically for split kernel and user virtual address spaces, with both a limit on the total of the two together (a 1G/3G split is common for Linux and BSD systems), and a limit on the visibility of physical memory over the 4G mark to a 32 bit kernel with a 4G limit on virtual address space, on a machine with >4G of physical RAM. In other words, it's a generic bounce buffer mechanism utilizing memory mapping windows to relocate so-called "high memory" into the kernel virtual address space so that the kernel can see it. It's unlikely that you will need to worry yourself with this sort of thing. In general, this is taken care of either by uio or by the IOMemoryDescriptor construct, which pretty much amount to the same thing. If the reason you are asking is that you are trying to find a kmap()- like mechanism to avoid polluting the VM/buffer cache with pages that you plan to use once and throw away clean (i.e. you are trying to write a streaming video server or something else that wants to move a lot of data, and since it isn't going to look at it again, it doesn't want to keep it around taking up space), then you should look at: and look at F_NOCACHE to disable caching of the data in the kernel by the unified buffer cache subsystem. If the reason you are asking is that you want to abuse the ability to map user space pages into the kernel virtual address space, an IOMemoryDescriptor will let you do this, in combination with the "map" member function, but I would discourage that, just like the Linux guys discourage using kmap() that way: This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Anton Altaparmakov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com What's the equivalent of Linux's kmap to map a user virtual address into the kernel memory space so that it is accessible "magically" (i.e. without any additional mapping/unmapping required) to all other tasks and kernel threads? What do you mean by 'accessible "magically"'? static inline struct page *ntfs_map_page(struct address_space *mapping, pgoff_t index) { struct page *page; page = read_mapping_page(mapping, index, NULL); if (!IS_ERR(page)) { wait_on_page_locked(page); kmap(page); if (PageUptodate(page) && !PageError(page)) return page; ntfs_unmap_page(page); return ERR_PTR(-EIO); } return page; } errno_t ntfs_page_map_ext(ntfs_inode *ni, s64 ofs, upl_t *upl, upl_page_info_array_t *pl, u8 **kaddr, const BOOL rw) { s64 size; kern_return_t kerr; int abort_flags; errno_t err; Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Mar 2007, at 11:32, Michael Smith wrote: On Mar 6, 2007, at 4:35 PM, Bhavesh Davda wrote: Are you just trying to make a portion of a user map visible in the kernel map, or are you trying to project that space into other maps as well? From what I can tell, the Linux 'kmap' call just maps a single user page somewhere into the kernel address space, so I'm assuming that you mean "so that other processes can call code in the kernel and see this page from there". No that is not kmap does. kmap is used to map the data content of a "struct page" (i.e. a page in the VM) into the kernel address space for access. I will give a concrete example from the Linux NTFS file system driver and show you the equivalent code from the OSX NTFS driver. If Bhavesh is asking about a file system then this is the answer he is looking for... Ok, on Linux in NTFS we map the content of a page at offset OFS of a file using (note index = OFS >> PAGE_CACHE_SHIFT): The important point of note is that once we have the struct page we run "kmap(page);" after which we can call "page_address(page)" to obtain the virtual memory pointer at which we can read/write the data content of the page. And now on OSX we do the same but it is a bit more involved (I have oversimplified the code for simplicity - we only really care about mapping the page anyway): if (ofs & PAGE_MASK) panic("%s() called with non page aligned offset (0x% llx).", __FUNCTION__, (unsigned long long)ofs); size = ubc_getsize(ni->vn); if (ofs > size) { ntfs_error(ni->vol->mp, "Offset 0x%llx is outside the end of " "the attribute (0x%llx).", (unsigned long long)ofs, (unsigned long long)size); return EINVAL; } /* Create a page list for the wanted page. */ kerr = ubc_create_upl(ni->vn, ofs, PAGE_SIZE, upl, pl, UPL_SET_LITE | (rw ? UPL_WILL_MODIFY : 0)); if (kerr != KERN_SUCCESS) panic("%s(): Failed to get page (error %d).\n", __FUNCTION__, (int)kerr); /* * If the page is not valid, need to read it in from the vnode now thus * making it valid. */ if (!upl_valid_page(*pl, 0)) { ntfs_debug("Reading page as it was not valid."); err = ntfs_pagein(ni, ofs, PAGE_SIZE, *upl, 0, UPL_IOSYNC | UPL_NOCOMMIT); if (err) { ntfs_error(ni->vol->mp, "Failed to read page (error " "%d).", err); goto pagein_err; } } /* Map the page into the kernel's address space. */ kerr = ubc_upl_map(*upl, (vm_offset_t*)kaddr); if (kerr == KERN_SUCCESS) { ntfs_debug("Done."); return 0; } ntfs_error(ni->vol->mp, "Failed to map page (error %d).", (int)kerr); err = EIO; pagein_err: abort_flags = UPL_ABORT_FREE_ON_EMPTY; if (!upl_valid_page(*pl, 0) || (vnode_isnocache(ni->vn) && !upl_dirty_page (*pl, 0))) abort_flags |= UPL_ABORT_DUMP_PAGES; ubc_upl_abort_range(*upl, 0, PAGE_SIZE, abort_flags); return err; } The important bit is ubc_upl_map() which maps all the pages in the UPL *upl into a linear virtual memory region which the start address of which will be returned in *kaddr. (i.e. *kaddr == page_address (page) on Linux). If you are not working with a file system then I would suggest checking out what ubc_upl_map() does and following it up that way to figure out what to do analogously. Typically, you don't want to map user data into the kernel in the first place; it's expensive and kernel VM is a scarce resource - typically you will find it's faster just to copy the page in/out. If you are trying to establish a shared page for the purposes of communication, I'd be inclined to encourage you to consider the system's native communication primitives instead. There are lots of ways of moving map entries around depending on what you're trying to achieve. I've already looked at vm_map and AFAICT, that ain't it. If you're doing this in an I/O Kit kext, the canonical tool for handling memory in another address space is the IOMemoryDescriptor. You can use this to map said memory, if you really need it. = Mike This email sent to site_archiver(a)lists.apple.com

1 0

Re: Cross Debugging
by Michael Smith 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 5, 2007, at 2:56 AM, Sven Mutke wrote: My question: is there a recommended way to set up Linux or FreeBSD to debug the kernel on the OSX machine? = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… No. Unless your goal is to get hopelessly frustrated debugging gdb, go buy an older Mac and use it to crash-test your kernel changes. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Michael Smith 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Mar 6, 2007, at 4:35 PM, Bhavesh Davda wrote: What's the equivalent of Linux's kmap to map a user virtual address into the kernel memory space so that it is accessible "magically" (i.e. without any additional mapping/unmapping required) to all other tasks and kernel threads? What do you mean by 'accessible "magically"'? = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Are you just trying to make a portion of a user map visible in the kernel map, or are you trying to project that space into other maps as well? From what I can tell, the Linux 'kmap' call just maps a single user page somewhere into the kernel address space, so I'm assuming that you mean "so that other processes can call code in the kernel and see this page from there". Typically, you don't want to map user data into the kernel in the first place; it's expensive and kernel VM is a scarce resource - typically you will find it's faster just to copy the page in/out. If you are trying to establish a shared page for the purposes of communication, I'd be inclined to encourage you to consider the system's native communication primitives instead. There are lots of ways of moving map entries around depending on what you're trying to achieve. I've already looked at vm_map and AFAICT, that ain't it. If you're doing this in an I/O Kit kext, the canonical tool for handling memory in another address space is the IOMemoryDescriptor. You can use this to map said memory, if you really need it. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Nikita Danilov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Organization: Cluster FS 07/03/2007 13:36 +0000, Anton Altaparmakov: > On 7 Mar 2007, at 07:52, Nikita Danilov wrote: > > 07/03/2007 04:37 -0800, Terry Lambert: > >> On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: > >> > >> [ ... ] > >> > > [...] > >>> (int)kerr); > >>> err = EIO; > >> > >> I may be misunderstanding the simplification you performed, and if > >> so, > >> forgive me... > >> > >> You want to call ubc_upl_map() before you do the I/O, or the UPL > >> could > >> have been recycled while the system was under extreme pressure, which > > > > By the way, is there any openly available documentation on UPL > > interface > > and XNU VM internals in general? > > - Really excellent book!: MAC OS X Internals by Amit Singh (http:// > www.amazon.co.uk/MAC-OS-Internals-Amit-Singh/dp/0321278542/) > > - http://developer.apple.com/referencelibrary/Darwin/idxKernel- > date.html and in particular the Kernel Programming Guide (http:// > developer.apple.com/documentation/Darwin/Conceptual/KernelProgramming/) Thanks, Anton, I'll take a look. > > - Download the xnu sources from http://www.opensource.apple.com/ > darwinsource/ and look in bsd/kern/ubc_subr.c. It has very good > comments above most of the function (kind of Kernel doc style like in > Linux but not quite). A couple of years ago, I used to read through osfmk/vm/ a lot, trying to understand UPL interface, hopefully it got better comments (read: any) since then. :-) > > Best regards, > > Anton Nikita. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Equivalent of Linux kmap on Mac OS
by Nikita Danilov 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Organization: Cluster FS 07/03/2007 04:37 -0800, Terry Lambert: > On Mar 7, 2007, at 3:45 AM, Anton Altaparmakov wrote: > > [ ... ] > [...] > > (int)kerr); > > err = EIO; > > I may be misunderstanding the simplification you performed, and if so, > forgive me... > > You want to call ubc_upl_map() before you do the I/O, or the UPL could > have been recycled while the system was under extreme pressure, which By the way, is there any openly available documentation on UPL interface and XNU VM internals in general? Nikita. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Equivalent of Linux kmap on Mac OS
by Bhavesh Davda 07 Mar '07

07 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=ETbziFjX+FcZROXwj08HcKII/wBBm6ZGAuP3xIZrvPju9m0hF6OUIRvRoI4wTiG46gK/fZdqhqGS3/iqBPiQApLhUgkdJQOxje4+gm3s4OARMMmEZQBds5Xfcr17l4xwybNMa9suky2ffpSj+qf5TeNYrGgTZylhPG0R+KKKOxU= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=kUpFGvMbL/VbDVSUFiooS25Y5QEuPscBibjhD+S8ljWnSSI6eUGkqUNB0/DhS6PpDAqfRBVBD3a/HMs641bZ6lkdOQX1wCrrHpDQlLaieXWsxErKcY8XCVBLdvh9jQtunnuJ0GpMM/wuMgrc1BoLsi6GyLW6+1qIDA4TsSFRebc= What's the equivalent of Linux's kmap to map a user virtual address into the kernel memory space so that it is accessible "magically" (i.e. without any additional mapping/unmapping required) to all other tasks and kernel threads? I've already looked at vm_map and AFAICT, that ain't it. Thanks -- Bhavesh P. Davda _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0