Re: Max number of pthreads in one process?
Re: Max number of pthreads in one process?
- Subject: Re: Max number of pthreads in one process?
- From: Army Research Lab <email@hidden>
- Date: Tue, 20 May 2008 15:17:25 -0400
- Thread-topic: Max number of pthreads in one process?
On 5/20/08 2:47 PM, "Terry Lambert" <email@hidden> wrote:
> On May 20, 2008, at 9:29 AM, Wade Tregaskis <email@hidden> wrote:
>
>>>> This intrigued me so I decided to try to track it down. Especially
>>>> since what I do see on Leopard is that the maximum number of threads
>>>> your demo app can create is 2559. When I looked into the kernel
>>>> code,
>>>> I found that unsurprisingly there's a compile-time limit on the
>>>> number
>>>> of threads per process, THREAD_MAX, which is 2560 in Leopard. When
>>>> this is hit thread_create_internal fails and returns KERN_FAILURE,
>>>> which I verified as best I can is what actually happens on my
>>>> system;
>>>> this propagates back all the way to userspace where it's
>>>> translated to
>>>> EAGAIN by the pthread library.
>>>
>>> OK, I don't have the sources here to look over, but that answers a
>>> different
>>> question for me; I was wondering if the machine type had something
>>> to do
>>> with how many threads you can spawn. My work machine is a dual-
>>> quad core
>>> machine, so I didn't know if that had something to do with it.
>>>
>>> Hmmm... actually... can someone with a dual-quad core machine running
>>> Leopard give that test app a quick whirl? If different systems get
>>> different results, then something VERY interesting is going on.
>>
>> I can reproduce this with 10.4.11 on my Quad G5. I cannot reproduce
>> it with Leopard on the same machine; it behaves exactly as on my
>> Intel machine under Leopard, which is to say it hits the 2560 limit
>> and fails gracefully.
>>
>> To track this down a bit, I took a System Trace of the app running
>> from start to finish (and moved it to an Intel machine so I could
>> analyse it in reasonable time... poor Quad's looking so old these
>> days). That confirmed what is actually already apparent if you look
>> at your output; although you're being told you're able to spawn 7k
>> threads, only 2559 ever actually run. In fact, System Trace tells
>> me that there are only ever 2560 threads in your app, total,
>> including your main thread. Unfortunately without Dtrace I can't
>> really poke into Tiger's kernel to see what's going on at runtime,
>> and I can't actually be sure System Trace is accurate in this case
>> because alas threads are identified via pointers, which are reused,
>> and if say there's a bug where new threads are clobbering old ones,
>> it's not guaranteed Shark will be able to see this.
>>
>> However, looking at the Libc source, I think I see the issue; in
>> Tiger's pthread implementation the pthread record (in userspace) is
>> allocated and configured before actually seeing if the corresponding
>> kernel pthread can be created; and even if that fails, it doesn't do
>> anything because it doesn't appear to be doing the necessary error
>> checking. :(
>>
>> It looks like the reason you eventually hit EAGAIN, nonetheless, is
>> that eventually it fails to preallocate the new thread's stack
>> space. I'm not sure why that has the odd but consistent ~7k limit;
>> I didn't investigate that.
>
> Try a 64 bit compilation of the app; in 32 bits, this is about where
> I'd expect your process virtual address space to be exhausted by stack
> allocation in the default size.
<<SNIP>>
Compiled and ran my program using gcc -m64 -Wall -Wextra -ggdb -std=c99.
The only difference is that now EAGAIN is never hit, and I'm still
'spawning' threads. As Wade noted, the maximum number of threads you can
actually spawn is 2560, but Libc will let you keep on allocating as if you
have resources for more. If you really want to watch this bog a system
down, run the following program (on a 64 bit system, and compiled using
-m64). The one truly significant change is that now I'm trying to spawn
100,000 threads. Under 10.4.11, I'm allowed to do so (I allocate the
resources for it in user space), but the kernel is only allows the first
2560 to run.
Thanks,
Cem Karan
#include <assert.h>
#include <errno.h>
#include <inttypes.h>
#include <limits.h>
#include <pthread.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <strings.h>
#include <unistd.h>
pthread_mutex_t myLock = PTHREAD_MUTEX_INITIALIZER;
bool dieNow = false;
void *threadFunc(void *arg)
{
printf("About to start loop.\n");
fflush(stdout);
while (true)
{
assert(pthread_mutex_lock(&myLock) == 0);
if (dieNow)
{
assert(pthread_mutex_unlock(&myLock) == 0);
break;
}
assert(pthread_mutex_unlock(&myLock) == 0);
usleep(100);
}
printf("Got message to quit.\n");
fflush(stdout);
return NULL;
}
int main(void)
{
pthread_t ID[100000];
uintmax_t counter, counter2;
int result;
bool noMoreThreads = false;
bzero(ID, 100000 * sizeof(pthread_t));
for (counter = 0; counter < UINT_MAX; counter++)
{
result = pthread_create(&(ID[counter]), NULL, threadFunc, NULL);
printf("Current number of threads is %" PRIuMAX "\n", counter);
fflush(stdout);
switch (result)
{
case 0:
break;
case EAGAIN:
printf("Maximum number of threads is %" PRIuMAX "\n",
counter);
fflush(stdout);
noMoreThreads = true;
break;
case EINVAL:
printf("How in the world did I get an EINVAL error???\n");
fflush(stdout);
exit(1);
break;
default:
printf("Unknown return value! %i \n", result);
fflush(stdout);
exit(1);
break;
}
if (noMoreThreads)
{
break;
}
}
sleep(1);
assert(pthread_mutex_lock(&myLock) == 0);
dieNow = true;
assert(pthread_mutex_unlock(&myLock) == 0);
for (counter2 = 0; counter2 < counter; counter2++)
{
printf("Quitting thread number %" PRIuMAX "\n", counter2);
fflush(stdout);
assert(pthread_join(ID[counter2], NULL) == 0);
}
return 0;
}
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden