May 2008 - Darwin-dev - lists.apple.com

Re: Max number of pthreads in one process?
by Wade Tregaskis 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Army Research Lab 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: Aci6cU5ujQSHPCZkEd2jIgAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 On 5/20/08 3:07 AM, "Wade Tregaskis" <wadeslists(a)mac.com> wrote: >> uname -a returns Darwin anvil.arl.army.mil 8.11.1 Darwin Kernel >> Version >> 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/ >> RELEASE_I386 i386 >> i386. This system is a fully patched 10.4.11 system. > > Although I too am curious what's up with this, I cannot reproduce > using 10.5.2 on my system. Have you tried on Leopard yourself? Note yet. This was just a quick hack to see what would happen, I'll play with it on another system that has Leopard. > In any case you should file a bug report (http://bugreport.apple.com/) > to ensure that the relevant people hear about your issue. I plan on it, but if its been fixed on Leopard, then it probably isn't something to worry too much about. > This intrigued me so I decided to try to track it down. Especially > since what I do see on Leopard is that the maximum number of threads > your demo app can create is 2559. When I looked into the kernel code, > I found that unsurprisingly there's a compile-time limit on the number > of threads per process, THREAD_MAX, which is 2560 in Leopard. When > this is hit thread_create_internal fails and returns KERN_FAILURE, > which I verified as best I can is what actually happens on my system; > this propagates back all the way to userspace where it's translated to > EAGAIN by the pthread library. OK, I don't have the sources here to look over, but that answers a different question for me; I was wondering if the machine type had something to do with how many threads you can spawn. My work machine is a dual-quad core machine, so I didn't know if that had something to do with it. Hmmm... actually... can someone with a dual-quad core machine running Leopard give that test app a quick whirl? If different systems get different results, then something VERY interesting is going on. > However, it looks like this is also what the limit was on Tiger (and > Panther, though on Jaguar it was 1024). So I'm curious how you're > able to create more threads than this? It doesn't look like it from > your uname output, but are you running a modified kernel? [[ It looks > like you could raise this limit easily enough by recompiling, but it > may not actually be safe to do so. ]] 100% plain vanilla kernel; whatever Apple ships with OS X 10.4.11 (with all of the most recent patches) is what I've got. OK, one other question then; does my test app exit normally on your machine? On mine, it always hangs (like I mentioned in my earlier post). The only way out is to kill the app. Thanks, Cem Karan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Wade Tregaskis 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Wade _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… uname -a returns Darwin anvil.arl.army.mil 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/ RELEASE_I386 i386 i386. This system is a fully patched 10.4.11 system. Although I too am curious what's up with this, I cannot reproduce using 10.5.2 on my system. Have you tried on Leopard yourself? In any case you should file a bug report (http://bugreport.apple.com/) to ensure that the relevant people hear about your issue. This intrigued me so I decided to try to track it down. Especially since what I do see on Leopard is that the maximum number of threads your demo app can create is 2559. When I looked into the kernel code, I found that unsurprisingly there's a compile-time limit on the number of threads per process, THREAD_MAX, which is 2560 in Leopard. When this is hit thread_create_internal fails and returns KERN_FAILURE, which I verified as best I can is what actually happens on my system; this propagates back all the way to userspace where it's translated to EAGAIN by the pthread library. However, it looks like this is also what the limit was on Tiger (and Panther, though on Jaguar it was 1024). So I'm curious how you're able to create more threads than this? It doesn't look like it from your uname output, but are you running a modified kernel? [[ It looks like you could raise this limit easily enough by recompiling, but it may not actually be safe to do so. ]] This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Eric Gouriou 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Diagnosing non-responsive Mac
by stephen joseph butler 19 May '08

19 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=ars8vTSO14HV3kcOQued4TmqvR1L9i8wxmtCJqtT9BY=; b=xDEyne6J+MNpoimm4PGU5wB3do7EC2ad229ae4vxZgf+Muc2Au2wMWkR3VztullpLH0K6ZIRb3TeV9m8Nc90bqfkgt+a5lZblM0a6/9Tb3EHrjyLpomaviGDeUJJ5oTZyZINXJP2kInxjkcUSXRwFrp+4b0VcZX4RHM08oGq2eQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=FYncHuYr1bi0E1jV0eKyfshkGc3gezW8qjobaAjjZ+JYxArSLLhcTP6bLQ07/FLxl2iwGB+IMyYfSmx1BTwhtYppDImVOb+HzgTTYxDaXZmFhWRlHIgJ0q9qJ4IRCj2xUVtzkfiSJXTi4mKwOgBiekp17t/qW3sruqARC7HNync= On Mon, May 19, 2008 at 5:05 PM, Andre-John Mas <ajmas(a)sympatico.ca> wrote: > Is there any way to diagnose what is causing a Mac to stop responding to > network access? I have a Mac that is using MacOS X server 10.4 and it > sometimes will quit responding to network request. When using the computer > directly loading an application will cause it to bounce for ever on the dock > or beach balling after certain operations. I've seen this before when some program on my system is forking new processes out of control/too rapidly. Just a thought. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Diagnosing non-responsive Mac
by Andre-John Mas 19 May '08

19 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi, ANdre _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Is there any way to diagnose what is causing a Mac to stop responding to network access? I have a Mac that is using MacOS X server 10.4 and it sometimes will quit responding to network request. When using the computer directly loading an application will cause it to bounce for ever on the dock or beach balling after certain operations. I changed the start-up to log into a user account and display CPU activity. When the machine stops being responsive I see no obvious peak in CPU. Is there any way to diagnose the issue of the problem? Maybe by use of a second Mac. The computer acting as server is a PowerMac G4. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Mike 19 May '08

19 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 2.0.0.14 (Macintosh/20080421) Hi all, just to see what would happen, I wrote up a small program that spawns as many threads as it can, and then tries to cancel all of them politely (see pasted code below). On my machine, I spawn 7051 threads before pthread_create() returns EAGAIN. This part is OK, I'm happy with it; its when I start canceling threads that I run into problems. Only the first 2559 threads are canceled. At that point, my main thread gets stuck in pthread_join() waiting for the other threads to cancel. Looking at it with Shark, I find I'm stuck in semaphore_wait_trap(). I don't have control over this, and as near as I can tell, I don't have any race conditions. So, where is the bug? uname -a returns Darwin anvil.arl.army.mil 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386. This system is a fully patched 10.4.11 system. The code was compiled with gcc -Wall -Wextra -pedantic -std=c99 -ggdb Thanks, Cem Karan #include <assert.h> #include <errno.h> #include <inttypes.h> #include <limits.h> #include <pthread.h> #include <stdbool.h> #include <stddef.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <unistd.h> pthread_mutex_t myLock = PTHREAD_MUTEX_INITIALIZER; bool dieNow = false; void *threadFunc(void *arg) { printf("About to start loop.\n"); Army Research Lab wrote: while (true) { assert(pthread_mutex_lock(&myLock) == 0); if (dieNow) { assert(pthread_mutex_unlock(&myLock) == 0); break; } assert(pthread_mutex_unlock(&myLock) == 0); usleep(100); } printf("Got message to quit.\n"); return NULL; } int main(void) { pthread_t ID[10000]; uintmax_t counter, counter2; int result; bool noMoreThreads = false; bzero(ID, 10000 * sizeof(pthread_t)); for (counter = 0; counter < UINT_MAX; counter++) { result = pthread_create(&(ID[counter]), NULL, threadFunc, NULL); switch (result) { case 0: break; case EAGAIN: printf("Maximum number of threads is %" PRIuMAX "\n", counter); noMoreThreads = true; break; case EINVAL: printf("How in the world did I get an EINVAL error???\n"); exit(1); break; default: printf("Unknown return value! %i \n", result); exit(1); break; } if (noMoreThreads) { break; } } sleep(1); assert(pthread_mutex_lock(&myLock) == 0); dieNow = true; assert(pthread_mutex_unlock(&myLock) == 0); for (counter2 = 0; counter2 < counter; counter2++) { printf("Quitting thread number %" PRIuMAX "\n", counter2); assert(pthread_join(ID[counter2], NULL) == 0); } return 0; } I believe I read somewhere the max # of processes per thread on OS X is 256, although you may want to check that. I think I recall reading it somewhere in the Multithreading Guide. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Max number of pthreads in one process?
by Army Research Lab 19 May '08

19 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: Aci54Mk7B4M2/CXUEd22gQAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 Hi all, just to see what would happen, I wrote up a small program that spawns as many threads as it can, and then tries to cancel all of them politely (see pasted code below). On my machine, I spawn 7051 threads before pthread_create() returns EAGAIN. This part is OK, I'm happy with it; its when I start canceling threads that I run into problems. Only the first 2559 threads are canceled. At that point, my main thread gets stuck in pthread_join() waiting for the other threads to cancel. Looking at it with Shark, I find I'm stuck in semaphore_wait_trap(). I don't have control over this, and as near as I can tell, I don't have any race conditions. So, where is the bug? uname -a returns Darwin anvil.arl.army.mil 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386. This system is a fully patched 10.4.11 system. The code was compiled with gcc -Wall -Wextra -pedantic -std=c99 -ggdb Thanks, Cem Karan #include <assert.h> #include <errno.h> #include <inttypes.h> #include <limits.h> #include <pthread.h> #include <stdbool.h> #include <stddef.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <unistd.h> pthread_mutex_t myLock = PTHREAD_MUTEX_INITIALIZER; bool dieNow = false; void *threadFunc(void *arg) { printf("About to start loop.\n"); while (true) { assert(pthread_mutex_lock(&myLock) == 0); if (dieNow) { assert(pthread_mutex_unlock(&myLock) == 0); break; } assert(pthread_mutex_unlock(&myLock) == 0); usleep(100); } printf("Got message to quit.\n"); return NULL; } int main(void) { pthread_t ID[10000]; uintmax_t counter, counter2; int result; bool noMoreThreads = false; bzero(ID, 10000 * sizeof(pthread_t)); for (counter = 0; counter < UINT_MAX; counter++) { result = pthread_create(&(ID[counter]), NULL, threadFunc, NULL); switch (result) { case 0: break; case EAGAIN: printf("Maximum number of threads is %" PRIuMAX "\n", counter); noMoreThreads = true; break; case EINVAL: printf("How in the world did I get an EINVAL error???\n"); exit(1); break; default: printf("Unknown return value! %i \n", result); exit(1); break; } if (noMoreThreads) { break; } } sleep(1); assert(pthread_mutex_lock(&myLock) == 0); dieNow = true; assert(pthread_mutex_unlock(&myLock) == 0); for (counter2 = 0; counter2 < counter; counter2++) { printf("Quitting thread number %" PRIuMAX "\n", counter2); assert(pthread_join(ID[counter2], NULL) == 0); } return 0; } _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Getting processor description from sysctl
by Terry Lambert 19 May '08

19 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On May 16, 2008, at 12:50 PM, "Jordan K. Hubbard" <jkh(a)apple.com> wrote: On May 16, 2008, at 10:29 AM, Finlay Dobbie wrote: It was intended to be a hook for SNMP. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… In this particular instance, it may very well be the case that parsing the XML output of system_profiler is the best option. My general point was that this is probably more fragile than a supported API would be, and it would be nice if the format of the XML (what keys there are, etc) was documented (like launchd.plist(5) for example) so there was at least some sort of vague nod from the "powers that be" that this is an OK thing to do - because, in general, parsing the output of the tools is regarded as fragile. ... And I think that's a perfectly fair request and certainly easier for us [Apple] than creating new API in those cases where the design and sustainability of said API isn't clear at all. Do you have a radar filed against the ASP folks to document the output of system_profiler(8)? While we're on the subject of API that isn't particularly well- designed or sustainable... If anyone has any ideas on how to replace (or supplement) sysctl(3) with something that overcomes all its obvious warts without also causing people to scream "Aaaaah! It's the Windows Registry! Ported over here to kill us all!", I'm all ears (or eyeballs). Terry and I are both old enough to remember the genesis of sysctl(3) in BSD as a "small number of simple knobs" mechanism and we both know that what we have today represents an over-usage of it to the point that it's been stretched well beyond its original design goals. It represents a problem to be fixed, and one that could conceivably be fixed in any number of ways, which is in itself a problem: Which is the "right" way? One obvious idea is to start with the new DSLocal store and make it a type of directory service, but there's a few non-obvious steps between there and step 3, Profit!, including the staying-away-from- the-Windows-Registry one mentioned earlier. Other, better, ideas may be less obvious to me. - Jordan This email sent to site_archiver(a)lists.apple.com

1 0

Re: Compiling CoreFoundation to link with CodeWarrior Windows?
by Jean-Daniel Dupas 19 May '08

19 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0