May 2008 - Darwin-dev - lists.apple.com

Re: malloc_zone_batch_malloc
by Ariel Burton 22 May '08

22 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mutt/1.4.1i The test program I included in my original psoting, however, tries all sizes from 1 bytes to 16mb without coming across a case where the routine returns a non-zero value. Am I missing something? Ariel Burton == On Wed 21 May 16:37, Terry Lambert wrote: > > On May 21, 2008, at 2:00 PM, Ariel Burton wrote: > >In what circumstances is malloc_zone_batch_malloc > >supposed to return any memory? > > <http://developer.apple.com/documentation/Performance/Conceptual/ManagingMem… > > > > "f your code allocates multiple, identically-sized memory blocks, you > can use the malloc_zone_batch_malloc function to allocate those blocks > all at once. This function offers better performance than the > equivalent series of calls to malloc to allocate the same memory. > Performance is best when the individual block size is relatively small— > less than 4K in size. The function does its best to allocate all of > the requested memory but may return less than was requested. When > using this function, check the return values carefully to see how many > blocks were actually allocated." > > > -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: malloc_zone_batch_malloc
by Terry Lambert 21 May '08

21 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com In what circumstances is malloc_zone_batch_malloc supposed to return any memory? -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On May 21, 2008, at 2:00 PM, Ariel Burton wrote: <http://developer.apple.com/documentation/Performance/Conceptual/ManagingMem… > "f your code allocates multiple, identically-sized memory blocks, you can use the malloc_zone_batch_malloc function to allocate those blocks all at once. This function offers better performance than the equivalent series of calls to malloc to allocate the same memory. Performance is best when the individual block size is relatively small— less than 4K in size. The function does its best to allocate all of the requested memory but may return less than was requested. When using this function, check the return values carefully to see how many blocks were actually allocated." This email sent to site_archiver(a)lists.apple.com

1 0

malloc_zone_batch_malloc
by Ariel Burton 21 May '08

21 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mutt/1.4.1i In what circumstances is malloc_zone_batch_malloc supposed to return any memory? Please try the program I reproduce below. Thanks, Ariel Burton == #include <stdio.h> #include <malloc/malloc.h> static size_t do_experiment ( malloc_zone_t *z, size_t size_bound ) { size_t ret_val = 0; void *v [ 2 ]; unsigned int err; size_t size; for ( size = 1; (! ret_val) && (size < size_bound); size++ ) { err = malloc_zone_batch_malloc ( z, size, v, 2 ); if ( 0 != err ) ret_val = size; } return ret_val; } /* do_experiment */ int main ( int argc, char *argv [] ) { size_t size; size_t size_bound = 16 * 1024 * 1024; malloc_zone_t *z = malloc_default_zone (); size = do_experiment ( z, size_bound ); printf ( "do_experiment to size %lu returned %lu\n", (unsigned long) size_bound, (unsigned long) size ); return 0; } _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Army Research Lab 21 May '08

21 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: Aci7L+iVJ2udsicjEd26hgAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 On 5/20/08 3:07 AM, "Wade Tregaskis" <wadeslists(a)mac.com> wrote: >> uname -a returns Darwin anvil.arl.army.mil 8.11.1 Darwin Kernel >> Version >> 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/ >> RELEASE_I386 i386 >> i386. This system is a fully patched 10.4.11 system. > > Although I too am curious what's up with this, I cannot reproduce > using 10.5.2 on my system. Have you tried on Leopard yourself? <<SNIP>> OK, I had a chance to test it last night on my home system (Intel Core Duo mac mini, running 10.5.2). It ran perfectly, so I'm guessing that the bug will get listed as fixed, and the code will be put into regression tests somewhere. Thanks, Cem Karan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Diagnosing non-responsive Mac
by Steve Sisak 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Is there any way to diagnose what is causing a Mac to stop responding to network access? I have a Mac that is using MacOS X server 10.4 and it sometimes will quit responding to network request. When using the computer directly loading an application will cause it to bounce for ever on the dock or beach balling after certain operations. I've seen this before when some program on my system is forking new processes out of control/too rapidly. Just a thought. -Steve _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… At 5:29 PM -0500 5/19/08, stephen joseph butler wrote: On Mon, May 19, 2008 at 5:05 PM, Andre-John Mas <ajmas(a)sympatico.ca> wrote: I think you're dead on -- this used to happen periodically with my dual G4/450 running Tiger server. In my case it was one of the windows sharing daemons running amok -- if you look in the logs you should see messages about the process table being full. I turned off Windows services and havne't had a problem since -- this may or may not be an option for you. You might try leaving a Terminal instance running, but IIRC, with the process table full you might not be able to execute shutdown or kill. I think I had to quit as much as possible and power cycle the machine to get it back. If you have the same problem and have Activity monitor running you can kill enough of them to be able to restart. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Compiling CoreFoundation to link with CodeWarrior Windows?
by Daniel G. Kluge 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Army Research Lab 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: Aci6r0wrivegQyaiEd2jIgAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 > ------------------------------ > > Message: 3 > Date: Mon, 19 May 2008 20:53:56 -0700 > From: Eric Gouriou <eric.gouriou(a)pobox.com> > Subject: Re: Max number of pthreads in one process? > To: Darwin Dev <darwin-dev(a)lists.apple.com> > Message-ID: <78202132-F85E-401E-91F2-CCDEBF848A94(a)pobox.com> > Content-Type: text/plain; charset="us-ascii" > > > On May 19, 2008, at 11:52 AM, Mike wrote: >> Army Research Lab wrote: > [... way too many threads ...] >> >> I believe I read somewhere the max # of processes per thread on OS X >> is 256, although you may want to check that. I think I recall >> reading it somewhere in the Multithreading Guide. > > It is 2500 on Leopard, per a recent discussion on the Java-Dev > mailing list. > > As in that discussion, I'll reiterate my (somewhat informed) opinion > that getting > anywhere close to that limit is a recipe for a performance disaster. > > Eric Don't worry, I'm not REALLY planning on doing anything like that! :) This was more of a 'what-if' exercise, just to see if any bugs turn up. And, as it turns out, one did. Thanks, Cem Karan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Army Research Lab 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: Aci6riK9YSu8vCahEd2jIgAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 On 5/20/08 2:47 PM, "Terry Lambert" <tlambert(a)apple.com> wrote: > On May 20, 2008, at 9:29 AM, Wade Tregaskis <wadeslists(a)mac.com> wrote: > >>>> This intrigued me so I decided to try to track it down. Especially >>>> since what I do see on Leopard is that the maximum number of threads >>>> your demo app can create is 2559. When I looked into the kernel >>>> code, >>>> I found that unsurprisingly there's a compile-time limit on the >>>> number >>>> of threads per process, THREAD_MAX, which is 2560 in Leopard. When >>>> this is hit thread_create_internal fails and returns KERN_FAILURE, >>>> which I verified as best I can is what actually happens on my >>>> system; >>>> this propagates back all the way to userspace where it's >>>> translated to >>>> EAGAIN by the pthread library. >>> >>> OK, I don't have the sources here to look over, but that answers a >>> different >>> question for me; I was wondering if the machine type had something >>> to do >>> with how many threads you can spawn. My work machine is a dual- >>> quad core >>> machine, so I didn't know if that had something to do with it. >>> >>> Hmmm... actually... can someone with a dual-quad core machine running >>> Leopard give that test app a quick whirl? If different systems get >>> different results, then something VERY interesting is going on. >> >> I can reproduce this with 10.4.11 on my Quad G5. I cannot reproduce >> it with Leopard on the same machine; it behaves exactly as on my >> Intel machine under Leopard, which is to say it hits the 2560 limit >> and fails gracefully. >> >> To track this down a bit, I took a System Trace of the app running >> from start to finish (and moved it to an Intel machine so I could >> analyse it in reasonable time... poor Quad's looking so old these >> days). That confirmed what is actually already apparent if you look >> at your output; although you're being told you're able to spawn 7k >> threads, only 2559 ever actually run. In fact, System Trace tells >> me that there are only ever 2560 threads in your app, total, >> including your main thread. Unfortunately without Dtrace I can't >> really poke into Tiger's kernel to see what's going on at runtime, >> and I can't actually be sure System Trace is accurate in this case >> because alas threads are identified via pointers, which are reused, >> and if say there's a bug where new threads are clobbering old ones, >> it's not guaranteed Shark will be able to see this. >> >> However, looking at the Libc source, I think I see the issue; in >> Tiger's pthread implementation the pthread record (in userspace) is >> allocated and configured before actually seeing if the corresponding >> kernel pthread can be created; and even if that fails, it doesn't do >> anything because it doesn't appear to be doing the necessary error >> checking. :( >> >> It looks like the reason you eventually hit EAGAIN, nonetheless, is >> that eventually it fails to preallocate the new thread's stack >> space. I'm not sure why that has the odd but consistent ~7k limit; >> I didn't investigate that. > > Try a 64 bit compilation of the app; in 32 bits, this is about where > I'd expect your process virtual address space to be exhausted by stack > allocation in the default size. <<SNIP>> Compiled and ran my program using gcc -m64 -Wall -Wextra -ggdb -std=c99. The only difference is that now EAGAIN is never hit, and I'm still 'spawning' threads. As Wade noted, the maximum number of threads you can actually spawn is 2560, but Libc will let you keep on allocating as if you have resources for more. If you really want to watch this bog a system down, run the following program (on a 64 bit system, and compiled using -m64). The one truly significant change is that now I'm trying to spawn 100,000 threads. Under 10.4.11, I'm allowed to do so (I allocate the resources for it in user space), but the kernel is only allows the first 2560 to run. Thanks, Cem Karan #include <assert.h> #include <errno.h> #include <inttypes.h> #include <limits.h> #include <pthread.h> #include <stdbool.h> #include <stddef.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <unistd.h> pthread_mutex_t myLock = PTHREAD_MUTEX_INITIALIZER; bool dieNow = false; void *threadFunc(void *arg) { printf("About to start loop.\n"); fflush(stdout); while (true) { assert(pthread_mutex_lock(&myLock) == 0); if (dieNow) { assert(pthread_mutex_unlock(&myLock) == 0); break; } assert(pthread_mutex_unlock(&myLock) == 0); usleep(100); } printf("Got message to quit.\n"); fflush(stdout); return NULL; } int main(void) { pthread_t ID[100000]; uintmax_t counter, counter2; int result; bool noMoreThreads = false; bzero(ID, 100000 * sizeof(pthread_t)); for (counter = 0; counter < UINT_MAX; counter++) { result = pthread_create(&(ID[counter]), NULL, threadFunc, NULL); printf("Current number of threads is %" PRIuMAX "\n", counter); fflush(stdout); switch (result) { case 0: break; case EAGAIN: printf("Maximum number of threads is %" PRIuMAX "\n", counter); fflush(stdout); noMoreThreads = true; break; case EINVAL: printf("How in the world did I get an EINVAL error???\n"); fflush(stdout); exit(1); break; default: printf("Unknown return value! %i \n", result); fflush(stdout); exit(1); break; } if (noMoreThreads) { break; } } sleep(1); assert(pthread_mutex_lock(&myLock) == 0); dieNow = true; assert(pthread_mutex_unlock(&myLock) == 0); for (counter2 = 0; counter2 < counter; counter2++) { printf("Quitting thread number %" PRIuMAX "\n", counter2); fflush(stdout); assert(pthread_join(ID[counter2], NULL) == 0); } return 0; } _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Terry Lambert 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Max number of pthreads in one process?
by Army Research Lab 20 May '08

20 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: Aci6my7sbXkAxiaOEd2jIgAdT0T19A== Thread-topic: Max number of pthreads in one process? User-agent: Microsoft-Entourage/11.4.0.080122 On 5/20/08 12:29 PM, "Wade Tregaskis" <wadeslists(a)mac.com> wrote: <<SNIP>> > > Again, though, file a bug report and cc me the number, so I can have > my sleuthing verified by those that actually do this stuff for their > day job. :) rdar://5948965 I have a feeling that this will get closed fairly rapidly though, after all, if its fixed in 10.5.2, why bother with it any more? Thanks, Cem Karan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0