June 2009 - Darwin-kernel - lists.apple.com

Re: Why support volfs ?
by Michael Smith 17 Jun '09

17 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why support volfs ?
by shailesh jain 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=xqnG2XfpCjzW1J4J5pFmiGE/qf9Tl0lJAVS9Vz+ryJA=; b=AG4it0eT5c0u+4sfb4WlwT3X7HQxcqCPVa95DJp1jqK1MBiKDVmnDChDrz6gRrBmRw McsyBHBlzsAs3LyzlBKszKZcJAc6prSFzInV3L4JOjWWkDUsBzTcoZG8ANs9oNcsKWSc tPCfqViRhrQ/BU4PtbrR7BXWxS0ElRMySNMSg= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=bFkCY8YEeKwC0VvHNSJZeRXnylDwHW6TXy+2oQF2vMnkI5uSbH5a51a6PrIv9qTMpW feBP5qY9XiLupj/6GIOh3JJtFe4RqA3LIHr1LjNoY3Zt//t/jLdObIJO+Fwq9Eoz3CQX ZKiySzlgvZd1EMNTtdqqFxrhn/ysLCZmoddEo= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why support volfs ?
by Michael Smith 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: malloc failure in a KEXT
by Michael Smith 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Why support volfs ?
by shailesh jain 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=c/lNICwE8NDW8P5HhX70qeloIaE8pey0PuC3ke9AUV8=; b=QYOz+9ovw9gkMoPSCgKM56/GSs4pKgjUs8Z4h74jksMiRpWY1I0Xhp1f7DRSaUkXem jiYjelBfCxtbC2QQyvtmZ7qzRG/7jK2gy91BgZRvqvzPLvxL913qm2JRZ+nK6hOg0XWg OU6frzXI3Clk0cMtkQlTeMrFU+kyxqNIUI7Qs= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=mVaDC6YCyIY42EAhiGs5Yshyh2MdWoyJrtjNWumLgUOxeZ9kFO/1ru1Afbti+pHIh3 V6xfjXXBwlyUzfXG6r6YVVBn/THAYbNt3EM7CnClkq/t+Grl5VfMWXpEggtmHfklLQra 4F0dcf9oVBg4h2+PpZ2oVofTfd/XVv8XP9n2k= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: malloc failure in a KEXT
by Joe van Tunen 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: Acnuv4y7urSh6Qz7Qq6gPSPe9N1YWQACeRcW Thread-topic: malloc failure in a KEXT User-agent: Microsoft-Entourage/11.4.0.080122 > From: Satyen Lele <SLele(a)Omneon.com> > Date: Tue, 16 Jun 2009 13:18:02 -0700 > > I am trying to debug a peculiar problem with malloc() failure inside a KEXT, > when it is trying to allocate a struct of 9184 bytes size. malloc() returns > ENOMEM after about 1139 such allocations that¹s just under 10MB. This > problem is only reproducible on the two Mac Pros with 2GB RAM and Mac OS X > 10.4.11. I panic¹ed the system in the KEXT just after malloc returned error. > Using zprint in gdb, I checked various memory zones and comparing the TOT_SZ > and MAX_SZ, I can see 1 zone to be running out of space. However, if ELT_SZ > is the allocation size of each element in the zone, it does not match the > allocation that is failing. > > ZONE COUNT TOT_SZ MAX_SZ ELT_SZ ALLOC_SZ NAME > 0x01f6ee40 2438 2bdfdc 2be000 44 > 1000 kernel map entries > > I have tried building the KEXT on a couple of Mac OS X and XCode combinations > 10.4.7+XCode2.5, 10.5.6+XCode3.1.2. > > How do I debug this problem? > Is there a way to find out which zone is being used for malloc()? > What do all the labels in zprint actually mean? > > Thanks, > -s. Looking at the source code for malloc would answer your questions I think. MALLOC calls kalloc which calls kalloc_canblock which calls kmem_alloc (kmem_alloc is called instead of zalloc_canblock because the requested size, 9184 bytes, is greater than 8192 bytes). zalloc_canblock works on a zone (one of the zones kalloc.1 to kalloc.8192) but kmem_alloc works on a vm_map called kalloc_map. There is a gdb command called zprint (from /Volumes/KernelDebugKit/kgmacros) which is similar to the shell command of the same name but it does not print fake zones such as kalloc.large (see /SourceCache/xnu/xnu-792.24.17/osfmk/kern/zalloc.c for a list of fake zones). kalloc.large is a fake zone representing the kalloc_map vm_map. A real zone has fixed sized elements. For the elem size column for kalloc.large, the shell zprint command prints the average size of the variable sized allocations. For the max size column, the shell zprint command prints the most bytes allocated for kalloc.large but for real zones it prints the maximum (that would be 16 MB for kalloc.large). The faked zone information for kalloc.large is provided by the kalloc_fake_zone_info function. That function uses the variables kalloc_large_inuse, kalloc_large_total, kalloc_large_max which all can be printed in gdb. The following command in gdb can print the entries of the vm_map used by kalloc.large: showmapvme kalloc_map This command will show the number of bytes used as greater than the number given by the zprint command (kalloc_large_total) because allocations are page aligned (4096 bytes per page) in vm_maps and the zprint info for kalloc.large does not take that into account like it does for real zones. You can modify showmapvme in kgmarcros to also print the sizes of free ranges. When you do that, you can see that even though more than 9184 bytes might be free, the largest free range is less than that and thus would cause MALLOC to fail. On Tiger (OS X 10.4) the kalloc.large vm_map has a hard limit of 16 MB compared to Leopard's (10.5) hard limit of 128 MB. 9184 bytes takes 3 pages = 12k. 12k * 1139 = 13.3 MB. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Understanding kexts in the early boot process
by Thomas Tempelmann 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=9e6+pRgwq+XHdH301uZh5t9sCfdsRxKRv/UvSzof7lU=; b=eLd42OdGuJt+TOhd8sutWzcxcgAfM0FvDmmMUB9rT5xT3tfve8Nk4kkuRb8hHKtvs0 WEx69Ke/2sHv7RO1P2HHMJHjCiVMDjeVeMYdzvViOb1HC5jLMq91HVNkVqCvA/VwgwzB aAi4tMS7Baoh5u0//VByTSGleVsnJRvjIa4UU= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=vP3dq8yGm+An10Cgnj74buDrBJSa3MWP3w2AMN9T3w5xfuezWvHKDpB+4/zTrv2AdC WL5gzojAV4+R4KDB8FomdDzj9Rivij1ixOyKSThRtEVO+RklKiX8jMFnqj2X6lKIdmR+ DV1XE3JXnA4cTnh5U0PyarLi8sK6n7AILNjsU= Andrew, David and Godfrey: Thank you all for the very detailed information. Together, this helped me understand how it all fits together. I am currently looking at boot loaders that boot Darwin on non-Apple hardware (mainly one called chameleon), where I saw the second set of kexts first. Now David mentions the very same technique (the "Extra" folder). Sadly, all the parts I've looked at so far hardly even give credit to this fact. I had seen David's name, but only in the source code of the boot loader itself. So, in regards to many people benefitting from this, let me thank you for your work on this, and for making it public :) -- Thomas Tempelmann, http://www.tempel.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

malloc failure in a KEXT
by Satyen Lele 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: Acnuv4y7urSh6Qz7Qq6gPSPe9N1YWQ== Thread-topic: malloc failure in a KEXT _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Understanding kexts in the early boot process
by Godfrey van der Linden 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Understanding kexts in the early boot process
by David Elliott 16 Jun '09

16 Jun '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jun 14, 2009, at 5:51 PM, Shantonu Sen wrote: On Jun 14, 2009, at 2:38 PM, Thomas Tempelmann wrote: This is definitely the file to start with. You'd probably have to ask the maintainer of that boot loader. You can find info (outdated, sorry) on it here: http://tgwbd.org/darwin/boot.html http://tgwbd.org/svn/Darwin/boot/branches/public/ Now, for a little bit of background. -Dave _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… What I haven't seen yet is the part that links the kexts (apparently loaded by the boot loader) to the kernel. Does the boot_args struct have a new parameter (not doc'd in Amit's book) which passes a pointer to a list of loaded kexts to the kernel, or how does this work? <http://opensource.apple.com/source/xnu/xnu-1228.12.14/libsa/catalogue.cpp > Background is this: It appears that a 3rd party boot loader is able to load two sets of kexts from different locations and get them used by the kernel. I try to understand how that works, because so far all I learned was that there's only one single group of kexts that the kernel sees, not several. Well, I'm not sure what bootloader exactly he's referring to, but I can be reasonably sure that it's a derivative of mine which is in turn a derivative of the traditional Darwin x86 "boot".[1] The public SVN tree is also available (can be used with SVN client or the latest revision can be browsed online): Another interesting resource is the BootX bootloader used on PowerPC Macs. As the Apple guys mentioned, the kext or mkext are loaded from disk (or network, but not working in "boot") by the bootloader be it BootX, boot, or boot.efi. The pointers to the in memory blobs are then passed into the kernel through entries in the device tree. These entries are the same regardless of platform. At any rate, the answer to the question is that the kernel doesn't really see it specifically as "two sets" of kernel extensions. If you look at libsa/catalogue.cpp you will find that it enumerates "/chosen/ memory-map" in gIODTPlane (the device tree plane is a plane that starts out initialized by the bootloader). A good place to start is the recordStartupExtensions function. For every child node whose name begins with "Driver-" it treats it as a kext bundle whose contents were read into memory by the bootloader and records it. For every child node whose name begins with "DriversPackage-" it treats it as an mkext bundle. This happens in a loop until all Driver-* and DriversPackage-* entries have been enumerated. So although normally the bootloader with either pass in one DriversPackage-* entry (i.e. Extensions.mkext was determined to be up to date) or multiple Driver-* entries (i.e. Extensions.mkext does not exist or was determined to be out of date) there is no specific requirement kernel-side that this be the case. You can mix and match all you want. In order to get OS X guest support working with Fusion we needed a way to add some extensions beyond those provided by Apple. Modifying the DVD was not even considered. The final product would have to be able to boot a retail boxed DVD. No tricks of any kind like modifying the filesystem behind the user's back. As I recall (and it's been a while now), one idea we almost settled on was to unpack the Extensions.mkext within the bootloader and pass the individual drivers to the kernel, removing some we didn't want, adding extras we did want, and modifying plists for some to make them drive more hardware[2]. I worked on this very early and at the time Fusion didn't even have a USB keyboard so if you wanted to type anything you needed PS/2 support in the guest. So as I began to research how I would implement this I took a look at the way the booter was passing the extensions in, both when it passed in a single mkext and also when it passed in individual drivers. I quickly released that essentially it was just forming a list of the in- memory blobs and passing pointers to the blobs to the kernel through the IODeviceTree. This led me to the libsa/catalogue.cpp file which revealed to me that the kernel is perfectly happy with any combination of kext (Driver-*) and mkext (DriversPackage-*). I realized I could pass in Extensions.mkext from the root device and still add in extra kexts or even a second mkext. All I needed to do was simply add the additional entries to the device tree. Of course using this method I could only add extra extensions. Modifying existing extensions would not be possible nor would removing them be possible. But removing them isn't really possible anyway if they are present on the root device because eventually kextd will run and will load them. I would say I probably spent more time discovering this useful tidbit of information than I actually spent implementing it. Once I knew how the kernel interpreted the information the implementation was really rather trivial. Hopefully that provides a little bit of background on how the extension loading process works and how extra extensions came to be. One thing to be aware of is that the bootloader does NOT do hardware detection to determine which extensions to load. As the Apple guys have mentioned, it uses OSBundleRequired.[3] This is buried in the source somewhere. The "Extra" extensions are loaded exactly the same way that the main ones are. So if .../Extra/Extensions.mkext is determined to be valid it loads. Otherwise .../Extra/Extensions is enumerated for kexts with an appropriate OSBundleRequired. That is in fact why it's Extra/Extensions and not ExtraExtensions or Extra_Extensions. It allowed me to use the same well-exercised code path. Personally I felt it was quite a hack to be able to load extra extensions beyond those present on the root device. But perhaps I need to choose my language more carefully and explain that that means I thought it was clever, lest I find myself in a room with an ignorant lawyer working for Apple asking me stupid questions again. Shakespeare was right. Footnotes: 1. Apple's boot in turn contains a fair amount of code very clearly derived from BSD and Mach. A fair amount of the support code hasn't changed in decades. 2. Somewhere around the time I figured out how to add extra kernel extensions I also figured out that the CFBundleIdentifier property of a personality can perfectly well refer to some other extension. So it became possible to add support for VMware's PIIX4 by putting the personality dictionary's from the Darwin release of AppleIntelPIIXATA into a new kext's Info.plist (one with a different top-level CFBundleIdentifier). By leaving the CFBundleIdentifier of the personalities themselves set to com.apple.driver.AppleIntelPIIXATA the kernel will load AppleIntelPIIXATA when one of these personalities matches. Assuming AppleIntelPIIXATA's probe method doesn't check for this case and fail probe. And assuming that it is capable of driving the hardware. This allowed me to add support for the PIIX4 by adding a new kext (with no real binary code) with the personalities rather than modifying the plist of the existing extension to add the personalities. 3. It's not particularly interesting except from an archeological standpoint, but there are remnants of code in the booter from an earlier time when it did apparently do some hardware detection to determine which extensions to load. It looks like this may have died with DriverKit which is the Objective-C precursor to the C++ IOKit. This email sent to site_archiver(a)lists.apple.com

1 0