January 2007 - Darwin-kernel

13 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kern_os_malloc?
by Michael Smith 13 Jan '07

13 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 12, 2007, at 8:08 PM, Greg wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Is there a performance hit to using the kern_os_malloc functions provided in the IOKit/pccard/k_compat.h header file (as opposed to OSMalloc)? Other than maybe another function call, probably not. The real question would be - why would you call it in the first place? This email sent to site_archiver(a)lists.apple.com

1 0

sshfs on Darwin with Userland Related Filesystem (urdfs)
by Takayama Fumihiko 13 Jan '07

13 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Thunderbird 1.5.0.9 (Macintosh/20061207) Dear All, I wrote "sshfs on Darwin" with Userland Related Filesystem (urdfs). http://mac.pqrs.org/sshfs/ Any comments appreciated. Regards, Takayama Fumihiko. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

kern_os_malloc?
by Greg 13 Jan '07

13 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com - Greg _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Is there a performance hit to using the kern_os_malloc functions provided in the IOKit/pccard/k_compat.h header file (as opposed to OSMalloc)? This email sent to site_archiver(a)lists.apple.com

1 0

Re: TCP, and clamping MSS..?
by Andrew Gallatin 12 Jan '07

12 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com John Heffner writes: > Andrew Gallatin wrote: > > Vincent Lubet writes: > > > Andrew, > > > > > > This is an optimization that is simple missin in xnu -- thanks for > > > bringing it to our attention. > > > > > > Feel free to file an enhancement request at http:// > > > bugreporter.apple.com/ to make sure this won't fall through the cracks. > > > > > > > Thanks Vince.. I've filed problem id 4919145. > > > > FWIW, it looks like the MSS clamping code was removed sometime between > > 10.2 and 10.3. I've never really understood why it helps so much. > > This isn't just my NIC that it helps for. See the thread starting at > > http://lists.freebsd.org/pipermail/freebsd-net/2004-January/002594.html > > where I actually proposed removing it, and found that it really did > > help! > > > > Do you have any intuition on why keeping things nicely aligned helps > > so much? Linux, on the same hardware, does not need to clamp the MTU, > > and easily maxes out the 10GbE link with far lower CPU utilization. > > Is the BSD stack just that much more sensitive to alignment? > > FWIW, Linux does exhibits this effect, too, though you probably won't > see it if it's doing TSO. I observed exactly this phenomenon on a > Pentium 3 back in the days when GigE was fast. I'm not up on the BSD > kernel, but in Linux, when using devices that support scatter/gather > DMA, packets are built as chains of pages. There's a certain amount of > allocation/copy overhead per page, so you get the best performance when > you just fill up 2 pages, and get a bit of a dip once you need 3. (This > might not be the same effect as in BSD, since it's using a constant > that's not page size...) I just tried with linux on the macpros (same machines), and with TSO disabled, do I see a consistent jump of 230Mb/s (9450 -> 9680) for a normal TCP_STREAM. For a zero-copy TCP_SENDFILE, it seems to actually hurt performance a little bit (9675 -> 9650). Neither is as pronounced as the 7000 -> 8000 jump I see on MacOSX. > If it goes back in, maybe it should have a sysctl? It's not clear this > optimization is always the right thing to do. There's a trade-off > between CPU efficiency and network efficiency which is a moving target > depending on the relative speeds of the CPU and NIC. When it's really > mattered for me, I've been satisfied just setting the interface MTU to > get the desired effect. I'd certainly be OK with a sysctl. I'm afraid that adjusting the MTU by hand may not work well, when you consider that the TCP (and IP) header size is not fixed. The common RFC1323 TCP timestamp option, for example, alters the header size by 12 bytes. This is why I'd prefer that the mss rounding code go back into the stack. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Understanding cores...
by Andreas Fink 11 Jan '07

11 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Understanding cores...
by Terry Lambert 11 Jan '07

11 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Jan 11, 2007, at 8:11 AM, Andreas Fink wrote: Typically, you just have to design your code so that it either fails safe, or it fails locally. If you end up corrupting memory, or end up walking off a pointer into a non existent address, or an address of something that used to be allocated, but is now freed, you will get either memory corruption (if you happen to hit something that's there, or if a freed area is reused for another purpose), or you crash with a fault in kernel mode. Thats nothing new for me. Actually most of the techniques you do there I already use in my own code. The KEXT we are looking at is however not developed by me but a bunch of very experienced kernel developers. It is very well working code written by very well experienced bsd developers (its actually code which is part of the OpenBSD kernel by now) and we seem to have a crash which is happen regularly on my setup but not on others. So we have some kind of race condition or so which is very hard to track down. And every time it goes down, we get 750MB of useless core dump unless someone can give me at least a glimpse if there's anything in there pointing us to the right direction somehow. I know this is a hard problem to track down but any hints are welcome. In the meantime we are continuing hunting improving the code on the way. We'll find it! My personal guess is that if it's an NKE, and the problem is only on Intel, you are running into a byte-order issue for trapping on the way in vs. out. Given the type of problem, I'd say that the issue is one of the packet header length field, rather than a checksum issue, so it's likely an IP header problem. But that's just a guess based on what you've said so far. We might also decide to boot with cpus=1 on an SMP system, to make sure that any locking we forget to do against reentrancy won't bite us. Happens on single CPU as well. That has been tested. XServe G4 single CPU, XServe Dual G5, MacPro 4xIntel, MacPro forced to single CPU, MacMini Dual CPU have all been seen crashing in the past. Nowadays the PPC's crash less than the intels but it has been opposite too a while ago (but we found a few other glitches on the way and fixed them too). We don't see any difference if the calls are synchronously, asynchronously, using poll or select, running in a single thread or multiple threads concurrently. Last traceback gave us some hint that it was somewhere in proto_delayed_inject() trying to lock a mutex (entry->domain->dom_mtx) If this is true, then it's not a CPU reentrancy into the code race, but it could still be a concurrency of access race, or it could be a timer event, etc., while something is blocked. It's likely something about persistent state being assumed not to change over a blocking operation. Then we are off to do the same under Leopard. (oh by the way, where can we checkout the sources of current leopard beta so we can verify our design?). You can't get the sources. We generally do not release xnu sources until after release, for reasons which should be obvious, if you think about them a little bit. If you need to test the KEXT and you are eligible for seeds (see <http://developer.apple.,com>), you can install one of the seed builds and load your KEXT into a binary seed kernel for testing. Well I have a binary of Leopard since WWDC2006. But that doesn't help for this KEXT as some internal structures are required to adapt it for Leopard. It loads fine but doesn't do its job. I've been asking Apple to make a public API out of it so we are not kernel version dependent but so far everyone is too busy to get Leopard out that it gets postponed all the time (I'm asking this since 2004 including personal visits to WWDC 2005 and WWDC 2006 and WWDC 2007 and...). >From my readings of the publicly visible responses in the radars you pointed to, it would be possible to implement the APIs yourself in Darwin and request their inclusion in MacOS X. This would also permit the near-term workaround of providing a replacement Darwin kernel for use by your customers as an interim solution until your APIs or variants end up getting adopted, or there's some other way made available to you to do the same thing. Anything else to do with thos radars probably needs handling in the context of those radars themselves. This email sent to site_archiver(a)lists.apple.com

1 0

Re: TCP, and clamping MSS..?
by John Heffner 11 Jan '07

11 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Thunderbird 1.5.0.9 (Macintosh/20061207) FWIW, it looks like the MSS clamping code was removed sometime between 10.2 and 10.3. I've never really understood why it helps so much. This isn't just my NIC that it helps for. See the thread starting at http://lists.freebsd.org/pipermail/freebsd-net/2004-January/002594.html where I actually proposed removing it, and found that it really did help! Do you have any intuition on why keeping things nicely aligned helps so much? Linux, on the same hardware, does not need to clamp the MTU, and easily maxes out the 10GbE link with far lower CPU utilization. Is the BSD stack just that much more sensitive to alignment? -John _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Andrew Gallatin wrote: Vincent Lubet writes: > Andrew, > > This is an optimization that is simple missin in xnu -- thanks for > bringing it to our attention. > > Feel free to file an enhancement request at http:// > bugreporter.apple.com/ to make sure this won't fall through the cracks. > Thanks Vince.. I've filed problem id 4919145. FWIW, Linux does exhibits this effect, too, though you probably won't see it if it's doing TSO. I observed exactly this phenomenon on a Pentium 3 back in the days when GigE was fast. I'm not up on the BSD kernel, but in Linux, when using devices that support scatter/gather DMA, packets are built as chains of pages. There's a certain amount of allocation/copy overhead per page, so you get the best performance when you just fill up 2 pages, and get a bit of a dip once you need 3. (This might not be the same effect as in BSD, since it's using a constant that's not page size...) If it goes back in, maybe it should have a sysctl? It's not clear this optimization is always the right thing to do. There's a trade-off between CPU efficiency and network efficiency which is a moving target depending on the relative speeds of the CPU and NIC. When it's really mattered for me, I've been satisfied just setting the interface MTU to get the desired effect. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Understanding cores...
by Michael Tuexen 11 Jan '07

11 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Terry, some additions: Best regards Michael On Jan 11, 2007, at 5:11 PM, Andreas Fink wrote: Andreas Fink Fink Consulting GmbH Global Networks Schweiz AG BebbiCell AG --------------------------------------------------------------- Tel: +41-61-6666330 Fax: +41-61-6666331 Mobile: +41-79-2457333 Address: Clarastrasse 3, 4058 Basel, Switzerland E-Mail: andreas(a)fink.org www.finkconsulting.com www.global-networks.ch www.bebbicell.ch --------------------------------------------------------------- ICQ: 8239353 MSN: msn1(a)gni.ch AIM: smsrelay Skype: andreasfink Yahoo: finkconsulting SMS: +41792457333 <GSM07-468-x-60.gif> _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… - it is not OpenBSD, but FreeBSD. The code is in the CVS repository of FreeBSD. - the SCTP implementation supports not only the RFCs mentioned below but also almost all SCTP extension currently described by Internet Drafts (and the intersection of the authors of the implementation and the authors of the RFC/ IDs is not empty). So this implementation is the one which supports more features than the Solaris or Linux one. - there are more users of the SCTP NKE: People use it for IPFix, MPICH supports now SCTP (also on Macs using the NKE), Reliable Server Pooling rsplib and more. None of them have the problems Andreas has. It is not clear what is so special with his setup. That is why we are asking if someone might have some suggestions to look at. Typically, you just have to design your code so that it either fails safe, or it fails locally. If you end up corrupting memory, or end up walking off a pointer into a non existent address, or an address of something that used to be allocated, but is now freed, you will get either memory corruption (if you happen to hit something that's there, or if a freed area is reused for another purpose), or you crash with a fault in kernel mode. Thats nothing new for me. Actually most of the techniques you do there I already use in my own code. The KEXT we are looking at is however not developed by me but a bunch of very experienced kernel developers. It is very well working code written by very well experienced bsd developers (its actually code which is part of the OpenBSD kernel by now) and we seem to have a crash which is happen regularly on my setup but not on others. So we have some kind of race condition or so which is very hard to track down. And every time it goes down, we get 750MB of useless core dump unless someone can give me at least a glimpse if there's anything in there pointing us to the right direction somehow. I know this is a hard problem to track down but any hints are welcome. In the meantime we are continuing hunting improving the code on the way. We'll find it! We might also decide to boot with cpus=1 on an SMP system, to make sure that any locking we forget to do against reentrancy won't bite us. Happens on single CPU as well. That has been tested. XServe G4 single CPU, XServe Dual G5, MacPro 4xIntel, MacPro forced to single CPU, MacMini Dual CPU have all been seen crashing in the past. Nowadays the PPC's crash less than the intels but it has been opposite too a while ago (but we found a few other glitches on the way and fixed them too). We don't see any difference if the calls are synchronously, asynchronously, using poll or select, running in a single thread or multiple threads concurrently. Last traceback gave us some hint that it was somewhere in proto_delayed_inject() trying to lock a mutex (entry->domain->dom_mtx) Then we are off to do the same under Leopard. (oh by the way, where can we checkout the sources of current leopard beta so we can verify our design?). You can't get the sources. We generally do not release xnu sources until after release, for reasons which should be obvious, if you think about them a little bit. If you need to test the KEXT and you are eligible for seeds (see <http:// developer.apple.,com>), you can install one of the seed builds and load your KEXT into a binary seed kernel for testing. Well I have a binary of Leopard since WWDC2006. But that doesn't help for this KEXT as some internal structures are required to adapt it for Leopard. It loads fine but doesn't do its job. I've been asking Apple to make a public API out of it so we are not kernel version dependent but so far everyone is too busy to get Leopard out that it gets postponed all the time (I'm asking this since 2004 including personal visits to WWDC 2005 and WWDC 2006 and WWDC 2007 and...). Unfortunately we simply don't have any other choice to implement this protocol. There is no other viable way to do this except throwing out MacOS X and go for Solaris (eiiik!)...or Linux (where is my user friendly GUI?) or OpenBSD (never touched so far) or Windows (please please not!) or do some really dirty hacking like implementing a layer 4 protocol on top of our own link layer (yes, we are so keen to implement a second IP stack...) or to implement it on a API built for IP filtering. MacOS X is the _LAST_ serious operating system I know which does not have SCTP (RFC3286, RFC2719,RFC 2960, RFC 3309, RFC 3257, RFC4460) yet... and SCTP is *MANDATORY* for any new Telecommunications protocol like M2PA (RFC4165), M3UA (RFC3332, RFC4666), SUA (RFC3868), IUA (RFC3057, RFC 3807, RFC4129, RFC 4233 ) which are the base of services like VoIP SS7 telephony, SIGTRAN, GSM network infrastructure, UMTS network infrastructure, international text messaging (SMS), Number Portability Systems and all the big iron Telco's have in their racks. Of course none of them today comes from Apple and Apple seems absolutely not interested to sell XServes into racks of telco giants having loads of money willing to spend millions of dollars on turnkey solutions provided by Apple developers. Conclusion: Apple is not considering telecommunications sector as being _ANY_ market for them ... ...and then Steve Jobs comes at MacWorld and is talking hours and hours about a pure telecommunications device...... (for Apple internals for the fun of it, read radar 4605154, 4609031, 4144183, 4674730) <GSM07-468-x-60.gif> _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/michael.tuexen% 40lurchi.franken.de This email sent to michael.tuexen(a)lurchi.franken.de This email sent to site_archiver(a)lists.apple.com

1 0

Re: TCP, and clamping MSS..?
by Vincent Lubet 11 Jan '07

11 Jan '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 11, 2007, at 6:16 AM, Andrew Gallatin wrote: We got it. Do you have any intuition on why keeping things nicely aligned helps so much? Linux, on the same hardware, does not need to clamp the MTU, and easily maxes out the 10GbE link with far lower CPU utilization. Is the BSD stack just that much more sensitive to alignment? Vincent _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Thanks Vince.. I've filed problem id 4919145. FWIW, it looks like the MSS clamping code was removed sometime between 10.2 and 10.3. I've never really understood why it helps so much. This isn't just my NIC that it helps for. See the thread starting at http://lists.freebsd.org/pipermail/freebsd-net/2004-January/ 002594.html where I actually proposed removing it, and found that it really did help! I'm not sure why it's so sensitive, may be it has something do with the mbuf allocator. This email sent to site_archiver(a)lists.apple.com

1 0