October 2007 - Darwin-kernel

Re: Leopard Sources?
by Chuck Remes 31 Oct '07

31 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Oct 30, 2007, at 11:04 PM, Chuck Remes wrote: I can't resist at least one more comment. cr _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… My recollection of the number of kernel patches discussed on the lists (darwin-dev, darwin-kernel and darwin-drivers) and submitted the last several years is... near zero. I agree there is a lot of irrational love for OSX by a lot of folks, but few of them are kernel hackers. Most of the folks who love OSX love it for the GUI and userland apps. The number of people really interested in the UNIX pieces of OSX couldn't even keep *one* darwin "distro" alive. Where's the love? Anyway, way off topic for the list. I just had to add to the noise... :-) If Andreas wants to add SCTP support to the kernel, I have a proposal. When the leopard sources appear (in a few days, weeks or months), do the hard work to create the KPI infrastructure you need to support this protocol. Write the code, create tests if possible along with a test harness (hard work for kernel code), plus documentation and submit it to Apple as a patch. Publish this patch far and wide, take feedback from other interested kernel hackers and *maintain* the patch as new Leopard releases ship. Make it your mission to get the patch accepted in 10.5.x or whatever the next major release is in 20 months. Don't wait for Apple to do this work for you (once you have the sources). Do the work, create the patch and be its champion. I think Apple folks would be surprised. Thus far the level of "community" support for kernel patches barely rises above *apathy*. Anyone who created a kernel patch and tried to support it this way would probably garner some sub-rosa engineering support within Apple simply because they'd be impressed with the effort. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Chuck Remes 31 Oct '07

31 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Oct 30, 2007, at 7:13 AM, Andrew Gallatin wrote: Quinn writes: [snip] Obviously it is possible to improve things in this space. For example, we could have a group of engineers dedicated to keeping the open source and the internal source in close sync. Doing this, however, would require Apple to rebalance its priorities: to increase our commitment to open source requires a corresponding decrease in the cool features that we can ship to customers (and developers). This makes no sense. The whole point of a software company going Open Source is to leverage the synergy of the community and allow your paid workforce to be more productive. Given many people's irrational love of MacOSX, I expect you'd gain lots of cool new features, and the main problem would be selecting the best ones. With the OpenSolaris development model, Sun provides an excellent example of how a traditionally closed source OS vendor has opened their development process. cr _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… My recollection of the number of kernel patches discussed on the lists (darwin-dev, darwin-kernel and darwin-drivers) and submitted the last several years is... near zero. I agree there is a lot of irrational love for OSX by a lot of folks, but few of them are kernel hackers. Most of the folks who love OSX love it for the GUI and userland apps. The number of people really interested in the UNIX pieces of OSX couldn't even keep *one* darwin "distro" alive. Where's the love? Anyway, way off topic for the list. I just had to add to the noise... :-) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Justin C. Walker 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Oct 30, 2007, at 13:49 , Anton Altaparmakov wrote: -rw-r--r-- 1 root wheel 4393120 Jun 24 12:27 mach_kernel -rw-r--r-- 1 root admin 8543400 Sep 24 2006 mach_kernel.orig You don't really need to use 'lipo' to see what the content is. Justin -- Justin C. Walker, Curmudgeon-At-Large Institute for the Enhancement of the Director's Income -------- Experience is what you get when you don't get what you want. -------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 30 Oct 2007, at 18:39, Rick Macklem wrote: On Tue, 30 Oct 2007, Michael Tuexen wrote: [snip] Just fyi, here's an "ls -l" for 10.4.9 on Intel. Despite the size difference, the kernel from sources seems to work fine on the hardware we have. (Does the vanilla one have more stuff linked in, instead of loading kexts? Haven't a clue.) [snip] Your comparison is flawed. You are comparing a single arch kernel (mach_kernel) with a bi-arch kernel (mach_kernel.org) Run: file /mach_kernel file /mach_kernel.orig Better yet, use 'size' or 'otool'. Just looking at the byte count on the disk won't show you what is going on inside the file, and for executables, I think you want to see that data. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Anton Altaparmakov 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I thought that they are about the same and therefore there can not be much difference in the source code files except for comments. -rw-r--r-- 1 root wheel 4393120 Jun 24 12:27 mach_kernel -rw-r--r-- 1 root admin 8543400 Sep 24 2006 mach_kernel.orig rick ps: I suspect mach_kernel.orig was actually an older 10.4 that 10.4.9, looking at the date on it. Run: file /mach_kernel file /mach_kernel.orig And you will see what I mean... To do the extraction do: sudo lipo /mach_kernel.orig -thin i386 -output /mach_kernel.orig.i386 Now repeat run: file /mach_kernel.orig.i386 Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 30 Oct 2007, at 18:39, Rick Macklem wrote: On Tue, 30 Oct 2007, Michael Tuexen wrote: I'm only interested in what is going in the file /mach_kernel. So are you saying that I'm not able to build a /mach_kernel file for the Intel platform using the available xnu sources for 10.4.8 which is equivalent to the /mach_kernel file Apple provides as part of 10.4.8? Just fyi, here's an "ls -l" for 10.4.9 on Intel. Despite the size difference, the kernel from sources seems to work fine on the hardware we have. (Does the vanilla one have more stuff linked in, instead of loading kexts? Haven't a clue.) Your comparison is flawed. You are comparing a single arch kernel (mach_kernel) with a bi-arch kernel (mach_kernel.org) A proper comparison would extract the single arch portion matching your architecture from the fat, original kernel and then you can compare the two sizes... And that show the same as "file /mach_kernel". And now you can compare the sizes properly... This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Rick Macklem 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Tue, 30 Oct 2007, Michael Tuexen wrote: I'm only interested in what is going in the file /mach_kernel. So are you saying that I'm not able to build a /mach_kernel file for the Intel platform using the available xnu sources for 10.4.8 which is equivalent to the /mach_kernel file Apple provides as part of 10.4.8? I thought that they are about the same and therefore there can not be much difference in the source code files except for comments. -rw-r--r-- 1 root wheel 4393120 Jun 24 12:27 mach_kernel -rw-r--r-- 1 root admin 8543400 Sep 24 2006 mach_kernel.orig rick ps: I suspect mach_kernel.orig was actually an older 10.4 that 10.4.9, looking at the date on it. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Just fyi, here's an "ls -l" for 10.4.9 on Intel. Despite the size difference, the kernel from sources seems to work fine on the hardware we have. (Does the vanilla one have more stuff linked in, instead of loading kexts? Haven't a clue.) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Andrew Gallatin 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Quinn writes: > I know of three main areas: > > o A significant chunk of what gets removed is comments. They might, <....> > o Some stuff is code that we received under non-disclosure from <....> > o Some stuff is removed because it relates to upcoming Apple Can somebody explain why the removal of the proprietary code cannot be automated? I get the feeling that there must be some poor guy going over the code by hand, line by line, and getting approval from a manager for each release. It is painfully obvious that this doesn't scale. > Having said that, Apple works hard to ensure that: > > a) Darwin can build and run as a standalone OS. > > b) You can slide a "mach_kernel" built from open source underneath > Mac OS X, and things still work. Not always. When the first Intel 10.4 kernels were released, compatibility was broken because some padding was removed from several key structures in in the C++ part of the kernel. This prevented you from running binary Kexts built on Darwin on MacOSX Intel, and vice-versa. I suspect this was meant to screw hackers who were running OSX on white boxes, but it also did a great job screwing people like me whose only way to test things involving PCIe prior to the release of the MacPro hardware was to use Darwin on a white box. I still have conditionals in my Makefiles to build for Intel darwin 8.0... <....> > Obviously it is possible to improve things in this space. For > example, we could have a group of engineers dedicated to keeping the > open source and the internal source in close sync. Doing this, > however, would require Apple to rebalance its priorities: to increase > our commitment to open source requires a corresponding decrease in > the cool features that we can ship to customers (and developers). > This makes no sense. The whole point of a software company going Open Source is to leverage the synergy of the community and allow your paid workforce to be more productive. Given many people's irrational love of MacOSX, I expect you'd gain lots of cool new features, and the main problem would be selecting the best ones. With the OpenSolaris development model, Sun provides an excellent example of how a traditionally closed source OS vendor has opened their development process. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Quinn 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com OK, I'll bite (-: I know of three main areas: - contain information related to upcoming hardware - explain workarounds for bugs in third party code, where Apple doesn't want to be seen as publicly criticising a third party - explain workarounds for Apple code, where Apple doesn't want to be seen as being critical of itself Having said that, Apple works hard to ensure that: a) Darwin can build and run as a standalone OS. * * * * * * <rdar://problem/3893228> Protocol plug-in KPI <rdar://problem/4605154> Requirement for SCTP implementation on Kernel * * * Share and Enjoy -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware At 21:00 -0600 29/10/07, William Kucharski wrote: It's not divulging trade secrets to give general areas which are/remain closed source. o A significant chunk of what gets removed is comments. They might, for example: o Some stuff is code that we received under non-disclosure from another party, or code based on information that we received under non-disclosure from another party. o Some stuff is removed because it relates to upcoming Apple products. For an extreme example, consider that, prior to the iPhone announcement, it would have been extremely foolish to reveal full support for the ARM platform. b) You can slide a "mach_kernel" built from open source underneath Mac OS X, and things still work. So, while the Darwin kernel source is slightly different from Apple's internal source, you'd be hard pressed to find a situation where that makes any difference in practice. If you discover such a situation, please let us know by filing a bug. At 18:30 -0600 29/10/07, William Kucharski wrote: The irony is that in PPC days the kernel was fully open sourced and it was only various libraries, such as the ones having to deal with DVDs, that were closed source. This is /not/ true. The situation I've described above has existing since the first Darwin release. Obviously it is possible to improve things in this space. For example, we could have a group of engineers dedicated to keeping the open source and the internal source in close sync. Doing this, however, would require Apple to rebalance its priorities: to increase our commitment to open source requires a corresponding decrease in the cool features that we can ship to customers (and developers). Decisions like that are made well above the pay grade of the folks that read this mailing list. To be blunt, complaining in this mailing list won't help. The Apple folks who read the list are just a bunch of kernel engineers. Regardless of whether we agree or disagree with you, it's not our decision to make. What you should do is corner an Apple marketing person [1] and explain to them the consequences of the current priorities. When you do that, avoid emotional appeals; rather, explain how the current policy negatively impacts Apple (either in dollar terms, or in terms of lost opportunity). With regards SCTP, we already have plenty of bugs covering this. The two lead bugs are: The first covers the requirement for a general protocol plug-in KPI, and the second covers the specific requirement for an SCTP implementation. Again, fixing this is a matter of priorities. In the three years since I filed <rdar://problem/3893228> I've encountered /two/ third party developers that need a protocol plug-in KPI: A. The first was commercial developer with a significant source base that was dead in the water on 10.4.x without this KPI. At the time this way a major kerfuffle. After a prolonged discussion, these folks eventually decided to change their in-kernel protocol code so that it was no longer an actual protocol (IIRC they hosted it on top of a UNIX domain socket, so the majority of their code could continue to use sockets for read, write, select and so on). They're now fully hosted on KPIs, and their code is more-or-less binary compatible with Leopard. B. The second is SCTP, as built by Michael and used by Andreas. This project continues to use the kernel's protocol plug-in mechanism, even though this isn't an official KPI. As such, it doesn't get the benefits that KPIs provide. That it, it is sometimes broken by dot releases of the OS, and it pretty much always broken by a major release. And it can't be fixed without Darwin source. Michael, Andreas, you have a decision to make. You can continue down your current path, with the negative consequences that I've described above. It's possible that Apple will eventually introduce a protocol plug-in KPI, but I can't say if or when that will happen. However, given that this is /not/ in 10.5, it seems likely that you'll be suffering this pain for years to come. Alternatively, you can find some other way to implement the functionality that you need. Perhaps you can move your protocol into user space? Or perhaps rehost your in-kernel code on top of UNIX domain sockets? I realise that this will be technically challenging. Ultimately, it's a question of balancing the pain of this change with the pain that you suffer by not being hosted on supported KPIs. I'm sorry to be so blunt about this. I'm sorry that we don't have a better story for you. I wish that we had infinite resources and could accommodate all of the needs of all of our developers. Alas, that just isn't the case. [1] Actually, Apple Developer Relations, and hence DTS, is part of marketing, but I pride myself is being the "most technical person in marketing" (-: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by William Kucharski 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Thunderbird 2.0.0.6 (X11/20070802) Michael Smith wrote: Really? William Kucharski _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… The irony is that this was not actually the case, but people didn't complain about it. As far as I know save for graphics drivers and various OS X libraries I never ran across anything that wasn't open source. That's not to say there wasn't of course, and I'm also not saying you can't build a functional xnu kernel from the current open source Intel code, but I'm curious as to what code might not have been complete then. It's not divulging trade secrets to give general areas which are/remain closed source. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Michael Smith 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Oct 29, 2007, at 4:59 PM, Michael Tuexen wrote: Things for which examples cannot be posted here. On Oct 29, 2007, at 5:30 PM, William Kucharski wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… There are many things in the kernel sources which are not meant for public viewing, and must first be removed. Once a new version has been posted and Could you give me an example of what kind of stuff "must first be removed"? The irony is that in PPC days the kernel was fully open sourced and it was only various libraries, such as the ones having to deal with DVDs, that were closed sourc The irony is that this was not actually the case, but people didn't complain about it. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leopard Sources?
by Michael Tuexen 30 Oct '07

30 Oct '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi William, I thought that they are about the same and therefore there can not be much difference in the source code files except for comments. Best regards Michael On Oct 30, 2007, at 1:30 AM, William Kucharski wrote: Michael Tuexen wrote: William Kucharski kucharsk(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I do know that the source code of a lot of kernel extensions/drivers/ whatever is not available from Apple and I'm not arguing about that. I'm only interested in what is going in the file /mach_kernel. So are you saying that I'm not able to build a /mach_kernel file for the Intel platform using the available xnu sources for 10.4.8 which is equivalent to the /mach_kernel file Apple provides as part of 10.4.8? There are many things in the kernel sources which are not meant for public viewing, and must first be removed. Once a new version has been posted and Could you give me an example of what kind of stuff "must first be removed"? the sources are subsequently available for download, the mailing list has been notified immediately thereafter, reliably. The irony is that in PPC days the kernel was fully open sourced and it was only various libraries, such as the ones having to deal with DVDs, that were closed source. I suspect since the switch to Intel, Intel has provided various proprietary drivers and board support code that Apple can't/won't release. Also, graphics drivers probably qualify both on PPC and Intel as ATI and NVidia are both pretty protective. This email sent to site_archiver(a)lists.apple.com

1 0