site_archiver@lists.apple.com Delivered-To: darwin-dev@lists.apple.com On Aug 30, 2007, at 12:04 PM, darwin-dev-request@lists.apple.com wrote: I've seen similar issues benchmarking 10GbE NICs, and I don't even need pthreads. The scheduler tends to run the user mode application on one core, the interrupt handler kernel thread (iokit "workloop") on another, and the network stack (dlil) kernel thread on yet another. II think the fundamental problem is the scheduler doesn't have a clue about cpu affinity, and MacOSX is lacking any APIs or command line interfaces that would allow the app or admin to give it a clue (like you can on Linux, Solaris, etc). Good would be a scheduler with some notion of CPU affinity, and better would be a scheduler that allowed the user to give it some hints. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.appl... Without wanting to rehash decades worth of research and hundreds of papers and experiments on the topic that comprise the body of experience on this subject, a couple of points are worth making. In and of itself, this is not an issue. More of a problem is that, in many cases, the threads you have noted above don't stay in the same cache domain. This *is* an issue. It is naive to assume, particularly in the context of a heavily threaded system like MacOS, that thread:CPU affinity will yield significant improvements here. The critical issue, as I note above, is not CPU affinity but rather cache affinity, and more specifically data:cache affinity, since the penalty is not cache load times as the threads move around (since the caches in question are typically large enough to hold the thread working set for each thread in play), but snoop/flush/reload time as dirty data moves (slowly) from one cache domain to another as it is handed off from one thread to the next. History suggests that this sort of hinting is a mixed blessing, often generating more problems than it solves (c.f. the NT stack binding vs. user comprehension for example). The real challenge involves solving, for some reduced set of circumstances, the very difficult but related questions: o What is the system going to do next? (e.g. should I schedule a thread that has just become runnable, or wait in the hope that the current thread will block soon?) o Who will want this data (that I have not yet examined or begun to process) next, and where are they now? (e.g. should I move this current thread to a different cache domain so that the consumer will not have to snoop it over?) In some restricted, single-activity cases, brute-force hinting approaches can help with the above. Sadly, hinting works much less well once you have more than one source of hints, or more than one topology in play, or a topology that is not understood by the hinter; an adaptive and automatic solution is much more attractive in that case. I don't think there's any disagreement that the current situation isn't very good; merely that the proposed 'solutions' fall fairly well short and that something better is really needed. This email sent to site_archiver@lists.apple.com