September 2007 - Darwin-dev

Re: Poor performance of pthreads across cpu dies.
by Andrew Gallatin 05 Sep '07

05 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Terry Lambert writes: > On Sep 4, 2007, at 6:09 AM, Andrew Gallatin wrote: > > The *NIC* has looked at it, and can easily hash different connections > > to several MSI-X interrupt handlers, which are then each bound to > > different CPUs (or groups of CPUs). > > I am told we support this (Message Signalled Interrupts) for the > latest Intel OS releases, and that it is in fact the preferred method > of interrupt delivery these days, for hardware capable of supporting > it. I wasn't able to find the tech note, which is supposed to have > been published already, according to the author, so you might want to > contact DTS. MSI is different from MSI-X. I figured out how to use MSI from the headers / source last winter, but AFAIK, darwin/macosx does not yet support MSI-X, which is required to have multiple interrupt vectors per device which are bound to different CPUs. In fact, I think MSI-X and support for multiple interrupt handlers per device is in direct conflict with the IOKit workloop design concept. If all the interrupt vectors activated the same workloop, MSI-X would be pointless. Having all the vectors serialized into a single workloop would loose any hope parallelism. And I think that a single driver is not allowed to have multiple workloops. > > Microsoft has though. It is called "receive side scaling" or RSS. > > The cleverness is that the NIC and the host has the connections to the > > same (sets of) CPUs. > > > > Even in the absence of RSS, multiple MSI-X interrupt handlers, etc, > > you can get a decent approximation of a crystal ball by having the > > ability to statically bind a single interrupt and a set of hot threads > > to set of CPUs. Eg, the administrator is the crystal ball. > > It's a little more complex than that when you throw power management > into the Mix, but it's more or less up to the driver author to do the > RSS implementation on top of MSI. Not MSI, MSI-X, which is AFAIK, missing in darwin/macosx. > Of course, this requires relatively smart hardware to take full I think all 10GbE NICs on the market today can do this, as well as some 1GbE NICs. > advantage of it (e.g. Microsoft want segment offloading in the cards, > plus scatter/gather, etc.). These features are supported in nearly every OS besides MacOSX. Heck, you can find TSO support on almost every 1Gbe NIC (including the Intel NICs built into Intel Apple hardware). I opened an enhancement request for TSO over 2 years ago (4131204). Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Localizing POSIX errno messages.
by Andre-John Mas 05 Sep '07

05 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 4-Sep-07, at 20:54 , James Farwell wrote: Greetings, A quick search turns up the following: Hopefully this is what you need. I haven't used this personally so I can't be of more help. Andre _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I have a standard POSIX errno (set by fopen, or malloc, etc.), and I am trying to turn it into a string localized for the current user. strerror() always seems to return the English string, no matter what I have done with setlocale(). I grepped around in the System folder and found that /System/Library/CoreServices/CoreTypes.bundle seems to have translations for all the POSIX error messages, but I can't find an API to get at them. Is there a System-level way to get at the localized version of these strings, or is it simply up to every application to define their own translations? http://developer.apple.com/documentation/Cocoa/Reference/Foundation/ Classes/NSError_Class/Reference/Reference.html This email sent to site_archiver(a)lists.apple.com

1 0

Localizing POSIX errno messages.
by James Farwell 05 Sep '07

05 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: AcfvV1msjKHQf2xOSD24Ss1h4V0gbg== Thread-topic: Localizing POSIX errno messages. Greetings, I have a standard POSIX errno (set by fopen, or malloc, etc.), and I am trying to turn it into a string localized for the current user. strerror() always seems to return the English string, no matter what I have done with setlocale(). I grepped around in the System folder and found that /System/Library/CoreServices/CoreTypes.bundle seems to have translations for all the POSIX error messages, but I can't find an API to get at them. Is there a System-level way to get at the localized version of these strings, or is it simply up to every application to define their own translations? Thanks! James _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Poor performance of pthreads across cpu dies.
by Terry Lambert 04 Sep '07

04 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com The *NIC* has looked at it, and can easily hash different connections to several MSI-X interrupt handlers, which are then each bound to different CPUs (or groups of CPUs). Microsoft has though. It is called "receive side scaling" or RSS. The cleverness is that the NIC and the host has the connections to the same (sets of) CPUs. Even in the absence of RSS, multiple MSI-X interrupt handlers, etc, you can get a decent approximation of a crystal ball by having the ability to statically bind a single interrupt and a set of hot threads to set of CPUs. Eg, the administrator is the crystal ball. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 4, 2007, at 6:09 AM, Andrew Gallatin wrote: I am told we support this (Message Signalled Interrupts) for the latest Intel OS releases, and that it is in fact the preferred method of interrupt delivery these days, for hardware capable of supporting it. I wasn't able to find the tech note, which is supposed to have been published already, according to the author, so you might want to contact DTS. It's a little more complex than that when you throw power management into the Mix, but it's more or less up to the driver author to do the RSS implementation on top of MSI. Of course, this requires relatively smart hardware to take full advantage of it (e.g. Microsoft want segment offloading in the cards, plus scatter/gather, etc.). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Poor performance of pthreads across cpu dies.
by Andrew Gallatin 04 Sep '07

04 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Michael Smith writes: > > On Aug 31, 2007, at 10:42 PM, Terry Lambert wrote: > > > I'd actually like to see an outside researcher demonstrate > > benchmarks vs. this approach. Mac OS X doesn't run in virtual wire > > mode for interrupt delivery, so interrupts are always fielded by the > > same processor. > > This is not necessarily the case. > > On some PPC platforms, interrupts are always handled by a single CPU. > > On the remainder, and on x86, there are several interrupt distribution > schemes in play. The x86 algorithm is (as I recall) roughly to prefer > CPUs that are awake, and of those prefer the CPU with the lowest APIC > ID that is not currently servicing an interrupt. > > As Drew points out, not a lot happens in interrupt context - the real > work is left to the scheduler invoking the workloop thread - but the > scheduler isn't terribly well off either. In the specific case of > network data input, the scheduler would need to know which userland > thread is currently blocked on (or will shortly read from) the socket > to which the data that the network adatper has just received will be > delivered. Never mind that no code has yet looked at this data, nor > that there may be data for several sockets/threads to be delivered. The *NIC* has looked at it, and can easily hash different connections to several MSI-X interrupt handlers, which are then each bound to different CPUs (or groups of CPUs). > Once it has worked this out, it needs to know which cache domain > currently contains the working set for these thread(s), and assess the > relative cost of moving them such that it can pick a domain in which > to run the workloop thread as well as any network-stack internal > threads or callouts. Sadly, Apple has not yet worked out a good > interface between the scheduler and commonly-available crystal balls, > and so this remains a difficult thing to do. Microsoft has though. It is called "receive side scaling" or RSS. The cleverness is that the NIC and the host has the connections to the same (sets of) CPUs. Even in the absence of RSS, multiple MSI-X interrupt handlers, etc, you can get a decent approximation of a crystal ball by having the ability to statically bind a single interrupt and a set of hot threads to set of CPUs. Eg, the administrator is the crystal ball. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Poor performance of pthreads across cpu dies.
by Michael Smith 01 Sep '07

01 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Aug 31, 2007, at 10:42 PM, Terry Lambert wrote: This is not necessarily the case. On some PPC platforms, interrupts are always handled by a single CPU. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'd actually like to see an outside researcher demonstrate benchmarks vs. this approach. Mac OS X doesn't run in virtual wire mode for interrupt delivery, so interrupts are always fielded by the same processor. On the remainder, and on x86, there are several interrupt distribution schemes in play. The x86 algorithm is (as I recall) roughly to prefer CPUs that are awake, and of those prefer the CPU with the lowest APIC ID that is not currently servicing an interrupt. As Drew points out, not a lot happens in interrupt context - the real work is left to the scheduler invoking the workloop thread - but the scheduler isn't terribly well off either. In the specific case of network data input, the scheduler would need to know which userland thread is currently blocked on (or will shortly read from) the socket to which the data that the network adatper has just received will be delivered. Never mind that no code has yet looked at this data, nor that there may be data for several sockets/threads to be delivered. Once it has worked this out, it needs to know which cache domain currently contains the working set for these thread(s), and assess the relative cost of moving them such that it can pick a domain in which to run the workloop thread as well as any network-stack internal threads or callouts. Sadly, Apple has not yet worked out a good interface between the scheduler and commonly-available crystal balls, and so this remains a difficult thing to do. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Poor performance of pthreads across cpu dies.
by Andrew Gallatin 01 Sep '07

01 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Terry Lambert writes: > On Aug 31, 2007, at 5:33 AM, Andrew Gallatin wrote: > > Michael Smith writes: > >> It is naive to assume, particularly in the context of a heavily > >> threaded system like MacOS, that thread:CPU affinity will yield > >> significant improvements here. > > > > It is perhaps more naive to presume that it won't. > > > > My 10GbE driver work makes me pretty familiar with the behavior > > under load of the network stacks in MacOSX, Linux, Solaris, > > and to a lesser Windows, all of which allow hinting. > > > > If you take another OS (say Linux), and intstall it on a Mac Pro, the > > only way to make it perform badly (like MacOSX) is to supply totally > > *incorrect* hints (bind the irq handler to one socket, and the > > benchmark to the other). And even then it does not perform as badly > > as MacOSX does on average :( If I then boot into OSX and apply the > > only sort of hinting I'm aware of -- disabling the second CPU package > > -- I get "reasonable" performance (2x the CPU utilization of Linux, > > and a hair less bandwidth). > > > > If I had a way to bind the ithread, the dlil thread, and > > the application to a CPU or set of CPUs, I could coerce > > MacOSX into getting decent performance without the drastic step > > of disabling a CPU core. > > I'd actually like to see an outside researcher demonstrate benchmarks > vs. this approach. Mac OS X doesn't run in virtual wire mode for > interrupt delivery, so interrupts are always fielded by the same > processor. You could maybe do something like the "poll mode" driver Yes, and no. The hardware interrupts are fielded by the same CPU, but I'm pretty sure the ithread, err, workloop can be scheduled anywhere. For an iokit network driver, the hardware interrupt context is nearly useless, and you have to be running in a thread context to pass packets up the stack. > in FreeBSD or like Rice Universities experimental LRP implementation > in NetBSD. Up to a point. But there's no way you could take it from > the interrupt all the way to the top of the stack, without taking the > I/O Kit model out of the picture (and the IOKit abstraction model buys > you a heck of a lot). So on average, you will still take 1.5 CPU > transitions for any workload on a 4 CPU system, even if you had a per- > CPU dlil thread. OSes with decent driver models (Solaris, FreeBSD, Linux) have no problem running from the interrupt (thread) all the way to the top of the stack and back down into the driver. So, I assume that iokit can't do this due to some deadlock around the workloop lock? Is this because there is some implicit aquisition of the workloop lock by iokit on the way to a transmit routine, or do some drivers explicity run the transmit routine in their workloop? Or is it some other deadlock? BTW, somebody needs to fix dlil and make the queue bounded. In a livelock situation, you can end up with all the mbuf clusters on the system sitting on the dlil input queue. The net.inet.ip.intr_queue_maxlen is a knob connected to nothing. > Most of the research in this area actually points to layering > violations at a very low level for specific protocols to force > classification of the data by destination process/thread. The > limitation on the supported protocols "undoes" the layering > violation. For systems supporting multiCPU and a large network device > buffer, this goes even further to tagging the descriptor by CPU, and > managing the connections as "named streams", where the interrupt is > sent (wherever). > > > Since you guys are fond of Solaris these days (Dtrace, ZFS, etc), I > > encourage you to take a look at the tools and APIs that Solaris > > provides (psrset(1M), processor_bind(2), pset_bind(2). > > IMO, the HP/IBM affinity interfaces are actually more likely to make > it through the standards process (they are already cross-vendor). It > would be a serious shame to implement one model, and then end up > having to throw it away because it wasn't the standard model, while > having to maintain both because someone wrote some important > application to the old one. Well, please just pick one and go with it. :) The more momentum it has behind it, the more likely it is to be a standard, and anything is better than nothing. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Poor performance of pthreads across cpu dies.
by Terry Lambert 01 Sep '07

01 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com It is naive to assume, particularly in the context of a heavily threaded system like MacOS, that thread:CPU affinity will yield significant improvements here. It is perhaps more naive to presume that it won't. My 10GbE driver work makes me pretty familiar with the behavior under load of the network stacks in MacOSX, Linux, Solaris, and to a lesser Windows, all of which allow hinting. If you take another OS (say Linux), and intstall it on a Mac Pro, the only way to make it perform badly (like MacOSX) is to supply totally *incorrect* hints (bind the irq handler to one socket, and the benchmark to the other). And even then it does not perform as badly as MacOSX does on average :( If I then boot into OSX and apply the only sort of hinting I'm aware of -- disabling the second CPU package -- I get "reasonable" performance (2x the CPU utilization of Linux, and a hair less bandwidth). If I had a way to bind the ithread, the dlil thread, and the application to a CPU or set of CPUs, I could coerce MacOSX into getting decent performance without the drastic step of disabling a CPU core. Since you guys are fond of Solaris these days (Dtrace, ZFS, etc), I encourage you to take a look at the tools and APIs that Solaris provides (psrset(1M), processor_bind(2), pset_bind(2). -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Aug 31, 2007, at 5:33 AM, Andrew Gallatin wrote: Michael Smith writes: I'd actually like to see an outside researcher demonstrate benchmarks vs. this approach. Mac OS X doesn't run in virtual wire mode for interrupt delivery, so interrupts are always fielded by the same processor. You could maybe do something like the "poll mode" driver in FreeBSD or like Rice Universities experimental LRP implementation in NetBSD. Up to a point. But there's no way you could take it from the interrupt all the way to the top of the stack, without taking the I/O Kit model out of the picture (and the IOKit abstraction model buys you a heck of a lot). So on average, you will still take 1.5 CPU transitions for any workload on a 4 CPU system, even if you had a per- CPU dlil thread. Most of the research in this area actually points to layering violations at a very low level for specific protocols to force classification of the data by destination process/thread. The limitation on the supported protocols "undoes" the layering violation. For systems supporting multiCPU and a large network device buffer, this goes even further to tagging the descriptor by CPU, and managing the connections as "named streams", where the interrupt is sent (wherever). IMO, the HP/IBM affinity interfaces are actually more likely to make it through the standards process (they are already cross-vendor). It would be a serious shame to implement one model, and then end up having to throw it away because it wasn't the standard model, while having to maintain both because someone wrote some important application to the old one. This email sent to site_archiver(a)lists.apple.com

1 0