March 2007 - Darwin-kernel - lists.apple.com

Re: xnu-792.18.15 source
by Michael Crawford 22 Mar '07

22 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Xdp30QzmVKhT8b6URW956bE+Sdw3zShoGFbQhS2CMM0HsiMQcxOqGeIpNBPbFhc/W7PukyPdfq1BoOnthchnOMxf7Dqc+Hf63kHkXrrmvZaaJBHxG06FMoFFKUPE9lqM2l867+KIhxB07rBkjVmMkmYgwx3tANUiPDV1XYKlMgE= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Z7XWJJcaehALTrUOxPyFUyVbHva7GR7pTE0tvIGCSuOMIqGja+TFHFt1dSPg1F/Bgb5/vzunn/x/cTwzTm9ZRXaafkK2eZUBQjEcdT16lR5W44KjdZsK0OFEFfAJMfcwc34Py7DLTpYSv96AyysJu6Dew/NNgRmfAsAsxNC7NH4= Does this mean that you can not recompile the sources and get a kernel that will function with all the usual Mac OS X applications? I am getting tired of poor filesystem performance and I was just thinking that I might spend a weekend fixing the obvious stuff in the VFS layer. No, it will work, and rebuilding the kernel is recommended by Apple when one wants to do source code debugging of drivers. I've been running a kernel I compiled myself on a Mac Pro for over a month now, without trouble. But there *are* some things that won't work. I just haven't run into any yet. Note that the kernel isn't just one single executable; the drivers are all separate modules. I think much of the code that Apple doesn't release is for some of the drivers. The kernel itself is mostly a hacked FreeBSD kernel combined with the IOKit and Mach. I've been reading the IOKit sources lately and it's all very straightforward stuff. I don't think there's much in the kernel itself (xnu, to be precise) that's likely to be under NDA. -- Michael David Crawford mdcrawford at gmail dot com Enjoy my art, photography, music and writing at http://www.geometricvisions.com/ --- Free Music Downloads --- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 3/22/07, Craig Newell <craign(a)ieee.org> wrote: This email sent to site_archiver(a)lists.apple.com

1 0

Re: xnu-792.18.15 source
by Craig Newell 22 Mar '07

22 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi There, You are making the assumption that: Thanks, CraigN _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com On Mar 22, 2007, at 17:10 , Michael Smith wrote: The source won't change after the binary is released that so it should be on the server right away. a) the source as released is necessarily exactly the source that was used to build the product b) the source can be packaged immediately for distribution in the format that it was in when the product was built Neither of these are true. In addition, you truly have no idea of the scope of the work involved to identify, scrub, validate and package the number of source code projects involved in a Darwin source release. Does this mean that you can not recompile the sources and get a kernel that will function with all the usual Mac OS X applications? I am getting tired of poor filesystem performance and I was just thinking that I might spend a weekend fixing the obvious stuff in the VFS layer. smime.p7s

1 0

Re: xnu-792.18.15 source
by Michael Smith 22 Mar '07

22 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Executable entry point
by Ronnie Misra 22 Mar '07

22 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Ronnie On Mar 22, 2007, at 2:39 AM, Jonas Maebe wrote: On 22 Mar 2007, at 10:23, Filipe Cabecinhas wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… You may be able to extract minimal bits of crt1.o to set up enough state to make dyld stubs work. However, once you start wanting to call out to libSystem, you'll probably need the crt externs to be set up (NXArgc/NXArgv, environ, etc.), and you'll need to call the appropriate Libc init code. You'll probably also have to properly align the stack. What I'm trying to suggest is that it's not worth duplicating all of crt1.o -- you should just link to the crt1.o already provided by the system, and have main call to your entry point. This suggests to me that dyld doesn't get initialised properly. Have you tried linking in /usr/lib/crt1.o already? In that case, you may not be able to specify an alternate entry point though. Another option may be to add the source code of crt1.o to your program so you can change its symbol names for entry point and where it jumps to in your program after it finishes initialising, but note that it was quite a convoluted mess the last time I looked at it. If it's publicly available could you show me that library so I could see how you've done that? Our source code is available, but I'm not sure if it'll be of much use of you since it's all written in Pascal (it's part of the run time library for the Free Pascal Compiler). If you want, I can point you to the files in question. But I'm not sure what you are referring to with "it". To clarify: I have not integrated the source code of crt1.o in our run time library. What we do is give our entry point the assembler name "_main" and link in crt1.o. crt1.o simply calls _main after it's finished with doing its thing, and the application takes over from there. This is the same way it works for gcc and (almost?) any other compiler which generates programs for Mac OS X. Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/rgmisra% 40apple.com This email sent to rgmisra(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Executable entry point
by Jonas Maebe 22 Mar '07

22 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 22 Mar 2007, at 10:23, Filipe Cabecinhas wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This suggests to me that dyld doesn't get initialised properly. Have you tried linking in /usr/lib/crt1.o already? In that case, you may not be able to specify an alternate entry point though. Another option may be to add the source code of crt1.o to your program so you can change its symbol names for entry point and where it jumps to in your program after it finishes initialising, but note that it was quite a convoluted mess the last time I looked at it. If it's publicly available could you show me that library so I could see how you've done that? Our source code is available, but I'm not sure if it'll be of much use of you since it's all written in Pascal (it's part of the run time library for the Free Pascal Compiler). If you want, I can point you to the files in question. But I'm not sure what you are referring to with "it". To clarify: I have not integrated the source code of crt1.o in our run time library. What we do is give our entry point the assembler name "_main" and link in crt1.o. crt1.o simply calls _main after it's finished with doing its thing, and the application takes over from there. This is the same way it works for gcc and (almost?) any other compiler which generates programs for Mac OS X. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Executable entry point
by Filipe Cabecinhas 22 Mar '07

22 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer; b=gQEOIs0NZgGZu6ZCEzdeIRIFP9V6A4NWzmeSaLM7kYp0WeH40dQLDQG4dIPaORo7kHCfglMj05Og0Bc1m8llDj2C0MShCVI3oWQu21MwERZSW6EwYIoBPQEKFlWQ2OksVSEocXIYCOGtGfNx9qOIyhtaUOv/CC0l6tZJVG3CrD4= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer; b=pyTHDQqqMrTr87UcsT2og8Yvd21BKd0gva2joXLletA3hq+DG4d6cR/ISnb8Usmk8L3MuI9HJYEVjJGiZKAHwIYzLDMS1MYYyGWTuDNof7dtQarQJv3b0/aso4Wbrp1Vnopx1ZIkzs9RYXKtDjcJd8Onl67/QtUZ8nUrYc4pORI= Thanks in advance Filipe Cabecinhas On 20 Mar, 2007, at 00:18, Jonas Maebe wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… If it's publicly available could you show me that library so I could see how you've done that? This suggests to me that dyld doesn't get initialised properly. Have you tried linking in /usr/lib/crt1.o already? In that case, you may not be able to specify an alternate entry point though. Another option may be to add the source code of crt1.o to your program so you can change its symbol names for entry point and where it jumps to in your program after it finishes initialising, but note that it was quite a convoluted mess the last time I looked at it. Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/filcab% 40gmail.com This email sent to filcab(a)gmail.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Executable entry point
by Terry Lambert 20 Mar '07

20 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Here it is: godel:~/dev/compiladores/tests filcab$ otool -l hello hello: [ ... ] It'll look something like this: Load command 6 cmd LC_LOAD_DYLINKER cmdsize 28 name /usr/lib/dyld (offset 12) -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Mar 20, 2007, at 4:13 AM, Filipe Cabecinhas wrote: On 20 Mar, 2007, at 10:07, Quinn wrote: Can you (Filipe) post the output from otool -l? Load command 8 cmd LC_UNIXTHREAD cmdsize 80 flavor i386_THREAD_STATE count i386_THREAD_STATE_COUNT eax 0x00000000 ebx 0x00000000 ecx 0x00000000 edx 0x00000000 edi 0x00000000 esi 0x00000000 ebp 0x00000000 esp 0x00000000 ss 0x0000001f eflags 0x00000000 eip 0x00002492 cs 0x00000017 ds 0x0000001f es 0x0000001f fs 0x00000000 gs 0x00000000 godel:~/dev/compiladores/tests filcab$ You should probably go to <http://developer.apple.com> and do a text search for "Mac OS X ABI Mach-O File Format Reference"; that said, knowing the object file format is not going to really be enough, without reading the rest of the ABI documentation. For example, there are several obvious problems with the binary you posted. Your thread state is corrupt; you should only set the eip; everything else should be zeroed. You are missing a LC_LOAD_DYLINKER section; the way the binary grading works, you need the right slice of the Universal dyld binary, and the only way to know that is to use the knowledge of the binary that you got from loading it - that's done in mach_loader.c, which loads up the dyld using a load command to set the path to it. The kernel really only knows about loading LC_SEGMENT, LC_SEGMENT_64, LC_UNIXTHREAD, and LC_LOAD_DYLINKER sections of binaries. Everything else is more or less ignored by the kernel, and dyld deals with it.. Depending on the specific tools you're using, you likely also need to load the library /usr/lib/libgcc_s.1.dylib before you load /usr/lib/ libSystem.B.dylib. I noticed that some of your data and text sections are improperly aligned; specifically, minimal alignment on some sections is 4, and __data and __bss need an alignment of 32 on Intel, since you might do an SSE spill or make SSE calls, and they have to be aligned to avoid your program dying as a result of an unaligned access fault. PPC needs an alignment of 4, in case of Altivec. It'd probably be informative for you to look at a binary that you already know that it works, e.g. "/bin/ls" or something similar and simple, and compare that to what you are generating. If you want to be a little more ambitious, then you could download the xnu source and look at the files in xnu/bsd/kern for mach_fat.c, mach_loader.c, mach_loader.h, and kern_exec.c. Oh, and you *really* don't want to be calling the system calls directly: they don't match the documentation, since the documentation for kill (for example) really documents the libc function; the actual kill system call takes a third argument to tell it whether or not you want it to act like the POSIX standard kill, or whether you ant it to act like the Legacy MacOS X call kill, which, when you e.g. send a signal to your own process group, whether or not you get the signal sent to you _by_ you depends on the third argument. And that's just one off the top of my head. The reason for this type of thing is that we define our system interface as being at the top of Libc, not between libc and the kernel. This lets us change the contract between the kernel and user space, if we need to to e.g. fix an issue, or provide a more efficient implementation under the covers, or just switch to an entirely new paradigm. It also lets us do things like make changes to e.g. the stat structure to return more information, and then hide the difference in a system call wrapper to maintain binary compatibility. Minimally, you will at least want to use our libc sources to learn the correct way to handle error returns and signals, etc.; the way error returns and signals are trampolined, for example, is to return to an alternate address - you don't necessarily come back to where you left, and if you don't e.g. fix up the thread state following a signal using the sigreturn entry point, you will quickly become sad every time someone sends you a signal (as in any time you reboot, for example). If you do decide to handle this all yourself anyway, then you'll definitely need the xnu and Libc sources, since the structures you pass into various Libc routines sometimes are not similar to the structures that pass the user/kernel boundary, and like I said earlier, there are argument list differences in the calls themselves. You'll need the internal versions of the structures that define what really goes across the user/kernel boundary. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Executable entry point
by Filipe Cabecinhas 20 Mar '07

20 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer; b=Ck7LyDARnsTcw7OBf9/n03BWg4gAve7ZBVjnnoCEXHbMwtEMqDZtTBJlB1JBGg73i8b9M8mlGFNEg2vQeFaAuFe2H5sFfCcByOEOq5R9JhxaeJcbpQhTfpea1fvPCEzUEh6S1aHoGqppBjk4mAZxRfryYbFjeau2xoiGl9xXFGs= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer; b=btsYm409/xGw5rUBYtXoUOYn7D5ECMlUQDKByj9Mce666Fkjd6rrp0yabO5ZmzFozb8/EtT8JefMLCdQvAH//9iKfiddOYnxYsqVWvLK0j6mGITubKC8II2DK5C8bAQX6ZiX3c9utRpxCBz9DXW9myD48Hpa3fijX23hBjnQ5/o= Here it is: Thanks for the help Filipe Cabecinhas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 20 Mar, 2007, at 10:07, Quinn wrote: Can you (Filipe) post the output from otool -l? godel:~/dev/compiladores/tests filcab$ otool -l hello hello: Load command 0 cmd LC_SEGMENT cmdsize 56 segname __PAGEZERO vmaddr 0x00000000 vmsize 0x00001000 fileoff 0 filesize 0 maxprot 0x00000000 initprot 0x00000000 nsects 0 flags 0x4 Load command 1 cmd LC_SEGMENT cmdsize 328 segname __TEXT vmaddr 0x00001000 vmsize 0x00002000 fileoff 0 filesize 8192 maxprot 0x00000007 initprot 0x00000005 nsects 4 flags 0x0 Section sectname __text segname __TEXT addr 0x000022f0 size 0x00000c87 offset 4848 align 2^0 (1) reloff 0 nreloc 0 flags 0x80000400 reserved1 0 reserved2 0 Section sectname __const segname __TEXT addr 0x00002f80 size 0x00000050 offset 8064 align 2^4 (16) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0 Section sectname __literal8 segname __TEXT addr 0x00002fd0 size 0x00000028 offset 8144 align 2^3 (8) reloff 0 nreloc 0 flags 0x00000004 reserved1 0 reserved2 0 Section sectname __textcoal_nt segname __TEXT addr 0x00002ff8 size 0x00000004 offset 8184 align 2^0 (1) reloff 0 nreloc 0 flags 0x8000040b reserved1 0 reserved2 0 Load command 2 cmd LC_SEGMENT cmdsize 260 segname __DATA vmaddr 0x00003000 vmsize 0x00001000 fileoff 8192 filesize 4096 maxprot 0x00000003 initprot 0x00000003 nsects 3 flags 0x0 Section sectname __const segname __DATA addr 0x00003000 size 0x0000000e offset 8192 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0 Section sectname __data segname __DATA addr 0x0000300e size 0x00000009 offset 8206 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0 Section sectname __bss segname __DATA addr 0x00003020 size 0x00000061 offset 0 align 2^5 (32) reloff 0 nreloc 0 flags 0x00000001 reserved1 0 reserved2 0 Load command 3 cmd LC_SEGMENT cmdsize 124 segname __IMPORT vmaddr 0x00004000 vmsize 0x00001000 fileoff 12288 filesize 4096 maxprot 0x00000007 initprot 0x00000007 nsects 1 flags 0x0 Section sectname __jump_table segname __IMPORT addr 0x00004000 size 0x0000000a offset 12288 align 2^0 (1) reloff 0 nreloc 0 flags 0x84000408 reserved1 0 (index into indirect symbol table) reserved2 5 (size of stubs) Load command 4 cmd LC_SEGMENT cmdsize 56 segname __LINKEDIT vmaddr 0x00005000 vmsize 0x00002000 fileoff 16384 filesize 6580 maxprot 0x00000003 initprot 0x00000001 nsects 0 flags 0x4 Load command 5 cmd LC_LOAD_DYLIB cmdsize 52 name /usr/lib/libSystem.B.dylib (offset 24) time stamp 1174303458 Mon Mar 19 11:24:18 2007 current version 88.3.6 compatibility version 1.0.0 Load command 6 cmd LC_SYMTAB cmdsize 24 symoff 16392 nsyms 400 stroff 21192 strsize 1772 Load command 7 cmd LC_DYSYMTAB cmdsize 80 ilocalsym 0 nlocalsym 372 iextdefsym 372 nextdefsym 25 iundefsym 397 nundefsym 3 tocoff 0 ntoc 0 modtaboff 0 nmodtab 0 extrefsymoff 0 nextrefsyms 0 indirectsymoff 16384 nindirectsyms 2 extreloff 0 nextrel 0 locreloff 0 nlocrel 0 Load command 8 cmd LC_UNIXTHREAD cmdsize 80 flavor i386_THREAD_STATE count i386_THREAD_STATE_COUNT eax 0x00000000 ebx 0x00000000 ecx 0x00000000 edx 0x00000000 edi 0x00000000 esi 0x00000000 ebp 0x00000000 esp 0x00000000 ss 0x0000001f eflags 0x00000000 eip 0x00002492 cs 0x00000017 ds 0x0000001f es 0x0000001f fs 0x00000000 gs 0x00000000 godel:~/dev/compiladores/tests filcab$ This email sent to site_archiver(a)lists.apple.com

1 0

Re: Apple Instrumentation libraray
by Quinn 20 Mar '07

20 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com <http://developer.apple.com/tools/download/> After that, check out the CHUD samples API in... CHUD.framework/Versions/A/Frameworks/CHUDCore.framework/Versions/A/Headers/chudSample.h <http://lists.apple.com/mailman/listinfo/perfoptimization-dev> S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… At 21:31 -0700 19/3/07, MFAWZY wrote: Does apple released a new library that replaced this functionality for OS X? Boy, that's a blast from the past. Unless you're working on traditional Mac OS, I'd recommend that you look at Shark. It can do a lot of cool stuff by itself, without modifying your code; and it has an API that lets you add custom instrumentation points. Curiously, the headers aren't installed when you install the CHUD framework as part of Xcode. You'll have to download CHUD directly. If you have questions about this stuff, you should ask them on the "PerfOptimization-dev" mailing list. That's where the real experts hang out. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Executable entry point
by Quinn 20 Mar '07

20 Mar '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Can you (Filipe) post the output from otool -l? S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… At 1:18 +0100 20/3/07, Jonas Maebe wrote: This suggests to me that dyld doesn't get initialised properly. Have you tried linking in /usr/lib/crt1.o already? That's definitely a good idea. However, dyld will bind the main executable to its imports before it calls the executable's entry point (that is, the PC stored in the LC_UNIXTHREAD command). My guess is that the Mach-O is not set up correctly to tell dyld to do this binding. However, it's hard to diagnose these sorts of problems without pouring through the Mach-O in detail. This email sent to site_archiver(a)lists.apple.com

1 0