site_archiver@lists.apple.com Delivered-To: darwin-dev@lists.apple.com Thread-index: Aci6riK9YSu8vCahEd2jIgAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 On 5/20/08 2:47 PM, "Terry Lambert" <tlambert@apple.com> wrote:

On May 20, 2008, at 9:29 AM, Wade Tregaskis <wadeslists@mac.com> wrote:

...

...

...

This intrigued me so I decided to try to track it down. Especially since what I do see on Leopard is that the maximum number of threads your demo app can create is 2559. When I looked into the kernel code, I found that unsurprisingly there's a compile-time limit on the number of threads per process, THREAD_MAX, which is 2560 in Leopard. When this is hit thread_create_internal fails and returns KERN_FAILURE, which I verified as best I can is what actually happens on my system; this propagates back all the way to userspace where it's translated to EAGAIN by the pthread library.

OK, I don't have the sources here to look over, but that answers a different question for me; I was wondering if the machine type had something to do with how many threads you can spawn. My work machine is a dual- quad core machine, so I didn't know if that had something to do with it.

Hmmm... actually... can someone with a dual-quad core machine running Leopard give that test app a quick whirl? If different systems get different results, then something VERY interesting is going on.

I can reproduce this with 10.4.11 on my Quad G5. I cannot reproduce it with Leopard on the same machine; it behaves exactly as on my Intel machine under Leopard, which is to say it hits the 2560 limit and fails gracefully.

To track this down a bit, I took a System Trace of the app running from start to finish (and moved it to an Intel machine so I could analyse it in reasonable time... poor Quad's looking so old these days). That confirmed what is actually already apparent if you look at your output; although you're being told you're able to spawn 7k threads, only 2559 ever actually run. In fact, System Trace tells me that there are only ever 2560 threads in your app, total, including your main thread. Unfortunately without Dtrace I can't really poke into Tiger's kernel to see what's going on at runtime, and I can't actually be sure System Trace is accurate in this case because alas threads are identified via pointers, which are reused, and if say there's a bug where new threads are clobbering old ones, it's not guaranteed Shark will be able to see this.

However, looking at the Libc source, I think I see the issue; in Tiger's pthread implementation the pthread record (in userspace) is allocated and configured before actually seeing if the corresponding kernel pthread can be created; and even if that fails, it doesn't do anything because it doesn't appear to be doing the necessary error checking. :(

It looks like the reason you eventually hit EAGAIN, nonetheless, is that eventually it fails to preallocate the new thread's stack space. I'm not sure why that has the odd but consistent ~7k limit; I didn't investigate that.

Try a 64 bit compilation of the app; in 32 bits, this is about where I'd expect your process virtual address space to be exhausted by stack allocation in the default size.

<<SNIP>> Compiled and ran my program using gcc -m64 -Wall -Wextra -ggdb -std=c99. The only difference is that now EAGAIN is never hit, and I'm still 'spawning' threads. As Wade noted, the maximum number of threads you can actually spawn is 2560, but Libc will let you keep on allocating as if you have resources for more. If you really want to watch this bog a system down, run the following program (on a 64 bit system, and compiled using -m64). The one truly significant change is that now I'm trying to spawn 100,000 threads. Under 10.4.11, I'm allowed to do so (I allocate the resources for it in user space), but the kernel is only allows the first 2560 to run. Thanks, Cem Karan #include <assert.h> #include <errno.h> #include <inttypes.h> #include <limits.h> #include <pthread.h> #include <stdbool.h> #include <stddef.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <unistd.h> pthread_mutex_t myLock = PTHREAD_MUTEX_INITIALIZER; bool dieNow = false; void *threadFunc(void *arg) { printf("About to start loop.\n"); fflush(stdout); while (true) { assert(pthread_mutex_lock(&myLock) == 0); if (dieNow) { assert(pthread_mutex_unlock(&myLock) == 0); break; } assert(pthread_mutex_unlock(&myLock) == 0); usleep(100); } printf("Got message to quit.\n"); fflush(stdout); return NULL; } int main(void) { pthread_t ID[100000]; uintmax_t counter, counter2; int result; bool noMoreThreads = false; bzero(ID, 100000 * sizeof(pthread_t)); for (counter = 0; counter < UINT_MAX; counter++) { result = pthread_create(&(ID[counter]), NULL, threadFunc, NULL); printf("Current number of threads is %" PRIuMAX "\n", counter); fflush(stdout); switch (result) { case 0: break; case EAGAIN: printf("Maximum number of threads is %" PRIuMAX "\n", counter); fflush(stdout); noMoreThreads = true; break; case EINVAL: printf("How in the world did I get an EINVAL error???\n"); fflush(stdout); exit(1); break; default: printf("Unknown return value! %i \n", result); fflush(stdout); exit(1); break; } if (noMoreThreads) { break; } } sleep(1); assert(pthread_mutex_lock(&myLock) == 0); dieNow = true; assert(pthread_mutex_unlock(&myLock) == 0); for (counter2 = 0; counter2 < counter; counter2++) { printf("Quitting thread number %" PRIuMAX "\n", counter2); fflush(stdout); assert(pthread_join(ID[counter2], NULL) == 0); } return 0; } _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.appl... This email sent to site_archiver@lists.apple.com