January 2003 - Darwin-kernel

Help recompiling xnu kernel.
by Mr. Wong 31 Jan '03

31 Jan '03

Hello everyone, I am new to Mac OS X and relatively new to unix environments. I am presently trying to recompile the Darwin Kernel (specifically the Mach kernel or XNU portion of Darwin) but I have noticed that the documentation has changed or is in a constant state of flux. Currently, I am running OS 10.2.3. Is it possible to rebuild the Mach Kernel with my OS or do I need to have an older (pre 10.2.3) OS to do it. If anybody out there has any suggestions I would appreciate any help. Thanks, Matthew K. Wong _________________________________________________________________ Protect your PC - get McAfee.com VirusScan Online http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963 _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Kernel -> user boundary
by Jim Magee 31 Jan '03

31 Jan '03

On Thursday, January 30, 2003, at 05:42 PM, Marek Kozubal wrote: I think this will do what you want. http://www.java-internals.com/code/mig_in_kernel.html Its a sample project on how to call MIG (Mach Interface Generator) functions from a kext. It passes the mach port in via sysctl, however you can also do it other ways (like one of your private args to the mount structure you pass mount()). I won't even mention what I think of its syscall-patching mechanism (except to say that it deserves to break at every opportunity). More specifically on this point, it makes a few incorrect assumptions about how the Darwin kernel works. First off: There really are two modes within the kernel. There is the kernel task mode - which is pretty much a full-blown Mach task for kernel-resident threads to execute within. Then there is the kernel's system/library executive mode - in which threads executes when they trap into the kernel from other task contexts. Not all kernel services work in both modes (many assume all callers are running in one mode or the other), and some services simply behave differently when called while running in the two modes. Mach IPC is one of those things that behaves quite differently, depending upon your kernel mode. When you are running in the trap mode, Mach assumes you are intimate with the implementation of Mach IPC (and may actually be assisting in its implementation). So lots of validity checking is bypassed. Port rights are held in limbo and not inserted into the kernel task's port name space (these are known as "naked" rights in the code comments) because Mach IPC assumes they are just on their way out to another task. Likewise, out-of-line memory regions are not mapped into the kernel's address space, but kept in vm_map_copy_t shorthand notation for later mapping into the true receiver task. Heck, even the Mach message headers are not converted into local perspective (local and remote ports swapped) because the message is just assumed to be "in-flight." Because of these distinctions, there are different IPC primitives available (send-from-kernel, rpc-from-kernel, but no "receive-from-kernel" because you aren't really a kernel thread at that point). Hooking up to receive trap-mode Mach IPC is even more difficult. MIG can compensate for some of these differences if you tell it you are running in the trap mode. But you have to deal with many of the differences manually (you ARE a kernel programmer after all...). In the example referenced here, they call mach_port_extract_right() from within trap mode. That routine expects to be running in that mode (which is good), but it expects to be doing its work on behalf of a user task, not the kernel. Specifically, the extracted port right is returned as a naked right - the caller is assumed to be a MIG-generated server-side stub which will pack it into a (trap mode) message and send that back out to user space. The port right is NOT inserted into the kernel ipc_space (port name space). All the calls trying to manipulate the kernel ipc_space are at best ineffective - leaking the naked port rights, and therefore the associated memory (~100000 exits and this thing will likely panic). Even worse, they are potentially more dangerous - by accidentally deallocating a right that really was inserted into the kernel's port name space by a kernel-bound thread (and causing it to misbehave in less obvious ways). See. None of this is easy. Is it? Programming unassisted in the kernel is not for mere mortals. That's why IOKit provides its own wrapper to all this that makes requesting and sending notifications, both to your own user-clients and to people listening on registry changes, relatively painless by comparison. As we develop facilities to support binary compatible extensions for other KEXT models (like FS, and NKEs), we will be providing notification mechanisms that match those models as well (with APIs that feel and behave natively to those programming environments and yet are catchable in other environments). This just takes quite a bit of time and effort to do right. But without it, we don't feel like we have a complete story and didn't want to have people trying the things that will clearly get them in trouble (like above). --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: filesystem port
by john spurling 31 Jan '03

31 Jan '03

firstly, my apologies for posting to the list twice. i didn't realize that the delay between sending and the message getting to the list was so long, so i sent it again. On Thu, Jan 30, 2003 at 10:57:28PM +0000, Quinn wrote: > At 10:16 -0800 30/1/03, john spurling wrote: > >however, i can't find anything in the way of documentation for > >vfs; > > There is very little Mac-specific documentation for VFS. There is > some generic kernel programming documentation on the Apple developer > web site. > > <http://developer.apple.com/techpubs/macosx/Darwin/index.html> yes, i've read good chunks of both the system overview and kernel programming books. > There are also some very specific Q&As. > > <http://developer.apple.com/qa/qa2001/qa1173.html> > <http://developer.apple.com/qa/qa2001/qa1113.html> ah, thanks; i'll check these out. > The source code to a variety of Apple file systems is part of Darwin. > > <http://www.apple.com/darwin/> i've checked the source out of xnu, but i was looking for any other high level docs before i start pouring through code. :) > That's about it as far as Apple resources are concerned. There are > some generic (and helpful) non-Apple VFS documents (the most relevant > being "The Design and Implementation of the 4.4BSD Operation System" > book), but these are not Mac OS X specific and don't cover Apple > additions (most notably, UBC). excellent! > >another piece of the puzzle is /System/Library/Filesystems. are these > >just kexts for the various filesystems? if so, why are they separate > >from the other kexts (in /System/Library/Extensions)? > > These items are used by autodiskmount to: > > 1. probe a disk to see whether it contains a specific file system, > 2. mount the file system, and > 3. fsck the file system. > > They are also used by Disk Utility to format new file systems. > > They are only relevant for local disk-based file systems. If you > write a local disk-based file system, you will have to add an item. > Network file systems work in a very different way. If you do a > network file system, you will probably want to plug-in to the > URLMount framework. I can help you with that offline. i'll definitely look into autodiskmount, then, as it's a disk based filesystem. > Unfortunately, all of this is undocumented )-: If you're doing a > commercial product, you might consider investing in some DTS tech > support incidents. no, this is something i'm doing on my own for fun. thanks so much for your help! -john _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Kernel -> user boundary
by Marek Kozubal 31 Jan '03

31 Jan '03

On Mon, 27 Jan 2003, [ISO-8859-1] George Andri wrote: > Hi there, > > I am aware of user -> kernel boundary crossing, like > Mach IPC, RPC, syscall, sysctl and ioctl. > > But I'm interested ina kernel -> user *initiated* connection, > is there anything outthere? I have a kext that needs > to initiate communication with an user process. I think this will do what you want. http://www.java-internals.com/code/mig_in_kernel.html Its a sample project on how to call MIG (Mach Interface Generator) functions from a kext. It passes the mach port in via sysctl, however you can also do it other ways (like one of your private args to the mount structure you pass mount()). You will probably be fine w/"simpleroutine"'s in the MIG files since you just need to fire off events and not wait for a reply/return. Unfortunately, it doesnt build out of the box for me, but the sample source is still very useful none the less. There are also tons of examples of MIGs thoughout the darwin sources. Just look for .defs files. --- Marek Kozubal marek(a)portents.com _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: filesystem port
by Quinn 30 Jan '03

30 Jan '03

At 10:16 -0800 30/1/03, john spurling wrote: i'm planning on porting a filesystem to darwin, and i have a few questions. i've read though the kernel programming guide, and making it a kext and coding against vfs seem like the right things to do. Yes. however, i can't find anything in the way of documentation for vfs; There is very little Mac-specific documentation for VFS. There is some generic kernel programming documentation on the Apple developer web site. <http://developer.apple.com/techpubs/macosx/Darwin/index.html> There are also some very specific Q&As. <http://developer.apple.com/qa/qa2001/qa1173.html> <http://developer.apple.com/qa/qa2001/qa1113.html> The source code to a variety of Apple file systems is part of Darwin. <http://www.apple.com/darwin/> That's about it as far as Apple resources are concerned. There are some generic (and helpful) non-Apple VFS documents (the most relevant being "The Design and Implementation of the 4.4BSD Operation System" book), but these are not Mac OS X specific and don't cover Apple additions (most notably, UBC). As with all development in the BSD portions of the Mac OS X kernel, binary compatibility is not guaranteed for VFS plug-ins. We probably won't document the VFS layer properly until we have the binary compatibility story sorted out. another piece of the puzzle is /System/Library/Filesystems. are these just kexts for the various filesystems? if so, why are they separate from the other kexts (in /System/Library/Extensions)? These items are used by autodiskmount to: 1. probe a disk to see whether it contains a specific file system, 2. mount the file system, and 3. fsck the file system. They are also used by Disk Utility to format new file systems. They are only relevant for local disk-based file systems. If you write a local disk-based file system, you will have to add an item. Network file systems work in a very different way. If you do a network file system, you will probably want to plug-in to the URLMount framework. I can help you with that offline. Unfortunately, all of this is undocumented )-: If you're doing a commercial product, you might consider investing in some DTS tech support incidents. <http://developer.apple.com/programs/techsupport.html> S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Fwd: Booting alternative kernels from primary disk?
by Walter Hop 30 Jan '03

30 Jan '03

> Here is a previous post about booting alternative kernels. You can > save yourself from the open firmware menu when you boot(see forwarded > message below). That was what I was looking for! > As for the most recent version of Darwin, this is a > regular question on this list :). In my opinion, Apple is slow to > update the public CVS repository to match their binary releases. Hmm, so by using CVS you actually get an older kernel. That's a disappointment. Well, I guess there is a reason for it. Hope it'll be updated to 10.2.3 soon :) thanks! walter -- Walter Hop <wh(a)slik.org> | +31 6 24452020 | http://www.skydancer.nl/ _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

filesystem port
by john spurling 30 Jan '03

30 Jan '03

greetings, i'm planning on porting a filesystem to darwin, and i have a few questions. i've read though the kernel programming guide, and making it a kext and coding against vfs seem like the right things to do. however, i can't find anything in the way of documentation for vfs; only the two header files in /System/Library/Frameworks/Kernel.framework/Headers/vfs. i've also browsed through the i/o kit documentation, but it doesn't look like i'll need anything there so far. am i right about this? another piece of the puzzle is /System/Library/Filesystems. are these just kexts for the various filesystems? if so, why are they separate from the other kexts (in /System/Library/Extensions)? the kernel programming guide gave me some good introductory info, but i think i'm looking for more specific information, like api docs. thanks, john _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

filesystem port
by john spurling 30 Jan '03

30 Jan '03

greetings, i'm planning on porting a filesystem to darwin, and i have a few questions. i've read though the kernel programming guide, and making it a kext and coding against vfs seem like the right things to do. however, i can't find anything in the way of documentation for vfs; only the two header files in /System/Library/Frameworks/Kernel.framework/Headers/vfs i've also browsed through the i/o kit documentation, but it doesn't look like i'll need anything there so far. am i right about this? another piece of the puzzle is /System/Library/Filesystems. are these just kexts for the various filesystems? if so, why are they separate from the other kexts? thanks, john _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Fwd: Booting alternative kernels from primary disk?
by Brian_Tabone 30 Jan '03

30 Jan '03

Walter, Here is a previous post about booting alternative kernels. You can save yourself from the open firmware menu when you boot(see forwarded message below). As for the most recent version of Darwin, this is a regular question on this list :). In my opinion, Apple is slow to update the public CVS repository to match their binary releases. My hunch is that they have new features they want to keep private until they release them to the general public, and so they don't keep the public CVS tree in sync with their local build. That's just me speculating though. Anyhow I think the latest darwin source you can get now is the 10.2.2 (tag xnu-344-12-2). -Brian Tabone Begin forwarded message: From: Josh de Cesare <decesare(a)apple.com> Date: Thu Jan 16, 2003 10:32:40 AM US/Central To: darwin-kernel(a)lists.apple.com Subject: Re: Booting alternative kernels from primary disk? Mark, Here's what I do after testing a kernel that did not work... Reboot, and stop at Open Firmware (hold CMD-OPT-O-F). Then I change the boot-file parameter to point at my back up kernel; usually /mach_kenrel.orig. Then boot the machine, fix the kernel and clear boot-file parameter. In OF type: printenv boot-device This will show the path to BootX; usually hd:9,\\:tbxi. The important thing to note here, is the partition number (9) and the path to the disk (hd:). When setting the boot-file use the same path and partition number, but put in the new kernel name at the end. setenv boot-file hd:9,mach_kernel.orig To continue booting type: boot After the machine has booted in terminal or on the console at root, fix /mach_kernel and clear the boot-file with the nvram command: nvram boot-fille="" What do you do if you mess up a boot driver on the disk? The only thing I have found is to boot off of a different partition and replace it with the original one. This can be done from single user off of an install CD. Here's the other way I do kernel or driver development... Net boot everything. It does require two machine and you will need a DHCP server on your network. First enable tftpd on your build machine. Copy /usr/standalone/ppc/bootx.xcoff to your tftpboot directory, usually /private/tftpboot. Make a link in /private/tftpboot to the build results of your xnu build and call is something like mach_kernel.test. You will also need to make an mkext to include all the important drivers. You will need a directory will all the drivers from the target build of the OS, I assume below that they are the same as the build machine. kextcache -lna ppc -m /private/tftpboot/mach_kernel.test.mkext /System/Library/Extensions The mkext name must be <kernel-name>.mkext Now the target machine needs to be configured. Set the boot-device to point over the network to BootX, and the boot-file to point at the kernel. Be sure to fill in the IP address of your build machine. setenv boot-device enet:12.34.56.78,bootx.xcoff setenv boot-file enet:12.34.56.78,mach_kernel.test At this point the machine should be able to boot, by finding BootX and the kernel and mkext, but it will not know what to do about the root disk. Set boot-args to point back at the disk. Normally, if it gets to single user, I'm done, so I include -s, and debug=0x14e in case it doesn't. setenv boot-args -s debug=0x14e rd=*hd:9 Once all the parameters are set type sync-nvram to flush them out. If the boot does not make it all the way through BootX, OF will not flush them out, and you will have to type it all again. As you rebuild new kernel, or change drivers (and rebuild the mkext) on your build machine all you need to do to test the new stuff is reboot the test machine. Good luck. Josh At 6:27 AM -0800 1/16/03, Mark Grimes wrote: "Before you install the new kernel, make a copy of your original kernel. When things fail you can always copy the original kernel back by booting from a different disk or partition. Either X or 9 will work." (from xnu building for mortals - circa 10.1.1) I am rather speechless. This seems like a rather obsolete suggestion for machines that ship with solely OSX now. Is there no such way to boot from the backup kernel you made from the primary disk?. I only have X, I don't have 9. I also only have one Mac. Do I need to keep the original install cd's on hand whenever I am in the mood to dabble? I have been searching the archives for this very question... either this is really a stupid question or everyone has enough methods to get themselves out of binds. Can someone give me a shout out as to where I missed the boat? -- Mark Grimes <mark(a)stateful.net> Stateful Labs _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Newbie- Keeping up with kernel development
by Walter Hop 30 Jan '03

30 Jan '03

Hi all, recently I "switched" my Intel desktop to Mac OS X. Coming from a FreeBSD background, the first thing I did was google for the way to checkout and build the Mach kernel from a latest snapshot. I carried out the procedure at http://www.opensource.apple.com/projects/documentation/howto/html/Building_… to build a new kernel on my Mac OS X desktop, but this raised a few questions. 1) The documentation on building your own mach_kernel seems a bit sparse, it felt a bit unsupported. Is it even considered useful to track the kernel on a Mac OS X box? Is there a mailinglist specifically for people doing this? 2) I checked out the source using 'cvs co -r Apple-201-5' as in the (mildly dated) HOWTO. This seemed to get me 6.2 which was to be expected from the old tag, so I tried 'cvs co -r HEAD xnu' to get a recent version. Strangely enough, this also looked like an older version than the regular /mach_kernel 6.3 that is supplied with Mac OS X 10.2.3: rez:src> strings xnu-HEAD/BUILD/obj/RELEASE_PPC/mach_kernel | grep "6\.3" rez:src> strings xnu-HEAD/BUILD/obj/RELEASE_PPC/mach_kernel | grep "6\.2" 'com.apple.kernel' = '6.2'; 'com.apple.kernel.bsd' = '6.2'; 'com.apple.kernel.iokit' = '6.2'; [...] rez:src> strings /mach_kernel | grep "6\.3" 'com.apple.kernel' = '6.3'; 'com.apple.kernel.bsd' = '6.3'; 'com.apple.kernel.iokit' = '6.3'; [...] I'd rather not run an older kernel than the stock kernel, does anyone have any information on how to get recent sources from CVS? Is there a STABLE-ish tag to test new stuff but stay exempt from too much breakage? :) Is it also possible to build the userland tools? 3) Most boot loaders such as FreeBSD's boot2 give you the option to boot a different kernel image by interrupting with some key. How can I save myself in Mac OS X on PPC hardware when I happen to boot into an invalid kernel? I found some info using a serial console, but that seems like a bit overkill... 4) I already have found a few gripes (mostly in mount_smbfs) that I can document fairly well, although I don't know enough about the kernel to create patches. What is the recommended procedure to report these half-hearted bug reports? Thanks in advance for the info :) walter -- Walter Hop <wh(a)slik.org> | +31 6 24452020 | http://www.skydancer.nl/ _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0