May 2008 - Darwin-dev - lists.apple.com

Re: Semaphored tasks not scheduling efficiently
by Wade Tregaskis 31 May '08

31 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Russ noted off-list that the problem was resolved by 10.5.3. Wade _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm not aware of anything that would cause the symptoms you're seeing that wasn't a result of contention; in the case where you have a core idle, what is your worker thread blocking on? "semaphore_timedwait_trap" is the mach incall that blocks a thread waiting for a semaphore wakeup, so you are blocking "legitimately". You might consider instrumenting your blocking points to see whether you spend an unexpectedly long time blocked at any given point - it sounds as though you expect to be running 8 threads non-stop, so it should be fairly straightforward to tell the difference between a legitimate and non-legit block. You should also check to make sure you aren't triggering any page-in/ out activity, as "tens of ms" is consistent with disk/network I/O timing. If you are handling multiple work units in a given run, it sounds like you might have a race between the work provider and consumers leading to a unit stuck in the queue. In the case where you have a thread asleep, have you checked the state of your work units? All this and more is provided in a System Trace, using Shark. Which Russ noted he already did. Without seeing the session, though, or at least the callstack for the blocking semaphore_timedwait_trap calls, there's not much point conjecturing. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Dealock issues related to mutex and thread termination.
by Terry Lambert 30 May '08

30 May '08

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com Hi, mach_port_t machThread = pthread_mach_thread_np(fThread); thread_terminate(machThread); So: (A) Give it the semaphore it wants (B) Signal it with pthread_kill() No. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On May 30, 2008, at 1:32 AM, Stéphane Letz wrote: We have a thread that locks a mutex and terminates, either itself or being "killed" by another thread. If this thread is "cancellable" using the phread API pthread_cancel()/pthread_join(), then we can possibly use pthread_cleanup_push/pthread_cleanup_pop at the appropriate place to properly unlock the mutex at termination stage. This seems to work correctly. But in some cases the thread is waiting on a low-level mach semaphore (we are directly using semaphore_wait/semaphore_signal kind of API) and the thread cannot be cancelled using pthread_cancel()/pthread_join(). Thus we use the following kind of code to terminate the thread: but then is seems that the pthread_cleanup_push/pthread_cleanup_pop method to unlock the mutex does not work anymore.... and we have a deadlock later on. 1) Is there a way to cancel a thread waiting using semaphore_wait using standard pthread API? So that thread_cleanup_push/ pthread_cleanup_pop can be used? 2) Is case 1) is no, is there any other method to add cleanup handlers when using low-level thread_terminate API? The pthread state is distributed between kernel (the backing Mach thread) and user space (the pthread itself). Cleanup handlers are on the pthread side. So if you terminate the kernel thread out from underneath the user space thread, it never returns to user space to permit the cleanup handlers to run to completion. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Semaphored tasks not scheduling efficiently
by Michael Smith 30 May '08

30 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On May 28, 2008, at 12:04 PM, darwin-dev-request(a)lists.apple.com wrote: Russ, = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I am optimizing a multithreaded app using shark on an 8-core machine. There is a typical MPCreateSemaphore/MPSignalSemaphore semaphore-based setup that hands out work units to be processed to a set of worker threads, and it is is working... but when you look carefully with Shark you see that a fair portion of the time, only 7 cores are busy. As the work units are being released, the cores start up --- but under close examination sometimes one will start only for 2 usec or less, then stop with semaphore_timedwait_trap. It then goes away for a long time (multiples of the 10msec os thread slice) and the mach kernel shows that there is an idle thread running. Sometimes the 'hole" in utilization moves to a different core. It winds up biting twice because not only do you lose a core, but you wind up with a ragged inefficient finish that wastes 6 or 7 cores. All this repeats, giving me a swiss-cheese look in what should logically be a very dense shark system trace. All the work eventually gets done and it works, but not as well as it should. I saw some notes on some mach semaphores that don't make the target task immediately runnable, not sure if that's what's happening here, but once the main thread goes to sleep after starting the workers, the eighth runnable worker thread should surely start, I'd think. Can anyone give insight into the MP semaphore and scheduling internals? Thanks. I'm not aware of anything that would cause the symptoms you're seeing that wasn't a result of contention; in the case where you have a core idle, what is your worker thread blocking on? "semaphore_timedwait_trap" is the mach incall that blocks a thread waiting for a semaphore wakeup, so you are blocking "legitimately". You might consider instrumenting your blocking points to see whether you spend an unexpectedly long time blocked at any given point - it sounds as though you expect to be running 8 threads non-stop, so it should be fairly straightforward to tell the difference between a legitimate and non-legit block. You should also check to make sure you aren't triggering any page-in/ out activity, as "tens of ms" is consistent with disk/network I/O timing. If you are handling multiple work units in a given run, it sounds like you might have a race between the work provider and consumers leading to a unit stuck in the queue. In the case where you have a thread asleep, have you checked the state of your work units? One final note; in a loosely-affine system like Darwin, I would tend to encourage a worker pool of N+(N/4) or so to ensure maximal saturation in the face of sporadic but protracted contention. This does have a tendency to give you a spiky tail on your saturation plot (since if contention is low you have a chunk of work remaining at the end that runs at about 1/4 saturation) so consider it no more than a starting point. Note that increasing the thread count into the 2-4N range can have useful effects on saturation depending on the nature of your workload, so do try moving both down and up. Obviously if you expect no contention, N is a good place to be. This email sent to site_archiver(a)lists.apple.com

1 0

Dealock issues related to mutex and thread termination.
by Stéphane Letz 30 May '08

30 May '08

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com Hi, mach_port_t machThread = pthread_mach_thread_np(fThread); thread_terminate(machThread); So: Thanks Stephane Letz _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… We have a thread that locks a mutex and terminates, either itself or being "killed" by another thread. If this thread is "cancellable" using the phread API pthread_cancel()/pthread_join(), then we can possibly use pthread_cleanup_push/pthread_cleanup_pop at the appropriate place to properly unlock the mutex at termination stage. This seems to work correctly. But in some cases the thread is waiting on a low-level mach semaphore (we are directly using semaphore_wait/semaphore_signal kind of API) and the thread cannot be cancelled using pthread_cancel()/ pthread_join(). Thus we use the following kind of code to terminate the thread: but then is seems that the pthread_cleanup_push/pthread_cleanup_pop method to unlock the mutex does not work anymore.... and we have a deadlock later on. 1) Is there a way to cancel a thread waiting using semaphore_wait using standard pthread API? So that thread_cleanup_push/ pthread_cleanup_pop can be used? 2) Is case 1) is no, is there any other method to add cleanup handlers when using low-level thread_terminate API? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Symbol not found in dylib
by Giuliano Gavazzi 30 May '08

30 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 30 May, 2008, at 07:20 , Phil Malin wrote: Hi all. dyld: Symbol not found: _environ Referenced from: /opt/local/lib/libfcgi.2.dylib Expected in: flat namespace Trace/BPT trap yes, in courier-authlib and some Apache httpd modules. See http://issues.apache.org/bugzilla/show_bug.cgi?id=4438 and the courier-imap mailing list. # The solution for apache was to use the latest GNU libtool: /usr/local/bin/libtool --version # ltmain.sh (GNU libtool) 1.5.26 (1.1220.2.492 2008/01/30 06:40:56) # Here is what I did for courier-authlib: # starting where the tarballs are going to be expanded: gnutar xjf courier-authlib-0.60.2.tar.bz2 cd courier-authlib-0.60.2 ./configure --without-authpam [...] # find and replace any occurrences of libtool in the build/source dir # MIND the following step runs ln with the force flag. Be sure to be in the correct directory! find . -type f -name libtool -print -exec ln -sf /usr/local/bin/ libtool \{\} \; make make check make install make install-configure # and authlib starts just fine. Giuliano _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Sorry if this question has been asked to death - I've done searches on it and have seen it referenced but haven't found any general discussion on the cause and solution. I've written a C program (well, an Eiffel program which compiles to C) which worked fine under Tiger but now fails under Leopard. The program links the fastcgi library libfcgi. When I try to run the program I get the following error: The above shows the MacPorts version - the same thing happens with the one in /usr/lib. I've even tried compiling the fastcgi source with my own compiler options and use that but to no avail. I'm guessing it has to do with osx' two level and flat namespace distinction and I have seen suggestions made in terms of compiler options but nothing I've tried seems to have helped. Has anyone else suffered this problem and if so, got a solution? I don't know if this has changed in new versions of 10.5, but this I report below is what I did for courier-authlib-0.60.2, hope this helps you. I your build does not use libtool it will not be of any use. Mind: you will have to install the latest libtool (probably not the latest by now...): This email sent to site_archiver(a)lists.apple.com

1 0

Symbol not found in dylib
by Phil Malin 30 May '08

30 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; bh=lT+FyEkb6Cs+8Lj2iECQEFWJZQMFFxMPcWZOx/kxowA=; b=ES1Smo6190E1F0o/WqVaHiBSRKfo+BHE67MRobQ+eeZLW1GwDIibPIgX3wfffWkqepRmu01IBZ9JM78tCrWioV/vUoTm88oybfK6WCl+CD33uN3+uTto1cMxj07VVJHlpDSx6Sn5NP4czc4fM3jTGQo1xGT4o2AtRBBJ83vqTGI= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; b=TO5Q5QbnUJNjryMZlk66qOU2IopIhatHONJSfEcNRfuPgn1MSQ0kk9hYhfUU6GOSfu80vhGdqyMmkysND0aV2ZosOmwEZUXRGITXw55xEcJHlLXdcDOREVTl48l7OLjRBJgk6WwZIFLEP66iFKDDxjhQse0g1h0q/maqC/sgWh8= User-agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.13) Gecko/20080412 SeaMonkey/1.1.9 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com dyld: Symbol not found: _environ Referenced from: /opt/local/lib/libfcgi.2.dylib Expected in: flat namespace Trace/BPT trap

1 0

Re: Error on leopard [hdiutil: attach failed - Resource busy ]
by spsaxena 28 May '08

28 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=8KKPQOxVQVPOmvxnZ8n3uHJwQlAvNVD4UFycVe9bG7Q=; b=aMGFW6lGRZEZ7rO0bMwUs753yvYOfYc12Ntv7fP9qVp/55UaTgE0FqGG1migZA1KsBFM+USV3B1GPKnapwag7zrNRZDzJ0n/8yFzC9zf0XGA4AkP9vvtjiz4mRD8EjJFb4S92Rs9WLti4X8xzfbqPNnuSFWzO0mncrMiE0p7+tk= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=aRx9ObpTwdeLO2Lpk8Dv2a38UjoBn38F+FR1XS+QdxAOT8LoAE7i/Ie+BdvPD1X2nnhM7dafcubvBrFKLI/pxqpBMt5Ju4zAmluGqFO+lhKlTmSpCkH4QadLk4FWi7UWDvDSgGWyWYOXtriVyoKgkzip+SZZ6pSiqFmp7Qv3/x8= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Error on leopard [hdiutil: attach failed - Resource busy ]
by spsaxena 28 May '08

28 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=fbrnWG0AWon18ylVtonC49HlyRpwUhWoHnSryHpKfZo=; b=GKJogI97oLu5QVYipon6TzWuC2ArdzjWomksnwlZ/NCyUKXJ2+LmQRCsl8g++2xCh0hBQNdxKlHYsEIljZEHGzpZN/NzhN9MMDse8OvMKsIA2FuKqQTY248DFwfY4QbPnxcSdCNhqiWpGkyKafKXKY3oMKClyUdT+l2mkgUJ8Ok= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=JTe4mBrUwz/bROJaenwnO+fFTBx6br/4/QGxEbcssiNzfMCJOa/rCYeNvsb3O6+MPBKAVRhxYmLPLFsWYVp47M3QQMmyRBISUw8gaqdnKKyWPxbm8yx9N0ZenjujAR0gk0rp24uoFHem9iI4DihB8eq10QnA8D4hX0SIgn+gaHg= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Error on leopard [hdiutil: attach failed - Resource busy ]
by Tim Schooley 28 May '08

28 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 2.0.0.14 (X11/20080513) Hi, -- $ hdiutil info or in plist format... $ hdiutil info -plist -- In the plist, you'll see an array entry something along the lines of: <dict> <key>content-hint</key> <string>Apple_HFS</string> <key>dev-entry</key> <string>/dev/disk1s2</string> <key>mount-point</key> <string>/Volumes/Qt 4.3.4</string> </dict> Hope this helps. Kind regards, Tim _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com spsaxena wrote: I am facing a problem on leopard with hdiutil. If I run the following commands then the second command fails as the dmg is already mounted *hdiutil attach -readonly -mountpoint /Users/test/AAA/ -plist /Users/test/test.dmg **hdiutil attach -readonly -mountpoint /Users/test/BBB/ -plist /Users/test/test.dmg* That is fine. But prior to OSX 10.5 the second command at least used to return me the plist where I can look for where the dmg was previously mounted. Now on leopard it does not return me the plist and just puts an error saying >>> hdiutil: attach failed - Resource busy which is of no use for me. Can you please let me know if there is anyway I can figure out where the dmg is mounted on leopard. I'm no expert on the subject, and may be wrong, but it appears that hdiutil has a command for listing currently mounted images: signature.asc

1 0

Semaphored tasks not scheduling efficiently
by Russ 27 May '08

27 May '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=d1OCJlyewcJ2QN3rjwxkmwaoTaKhdMk0ousNvMadASCvZpWZYb2FkgLAeKKbmWJ4UYEumUyootqea3bY0sHSGjdbKv3mexxb5j7llDbYuGafb00VRpgC2Bvz3Hl6wk2RScyrG6SkhOWyIWSFyaumZbkDMHqPwQWt8Y7l6gN2uZI=; I am optimizing a multithreaded app using shark on an 8-core machine. There is a typical MPCreateSemaphore/MPSignalSemaphore semaphore-based setup that hands out work units to be processed to a set of worker threads, and it is is working... but when you look carefully with Shark you see that a fair portion of the time, only 7 cores are busy. As the work units are being released, the cores start up --- but under close examination sometimes one will start only for 2 usec or less, then stop with semaphore_timedwait_trap. It then goes away for a long time (multiples of the 10msec os thread slice) and the mach kernel shows that there is an idle thread running. Sometimes the 'hole" in utilization moves to a different core. It winds up biting twice because not only do you lose a core, but you wind up with a ragged inefficient finish that wastes 6 or 7 cores. All this repeats, giving me a swiss-cheese look in what should logically be a very dense shark system trace. All the work eventually gets done and it works, but not as well as it should. I saw some notes on some mach semaphores that don't make the target task immediately runnable, not sure if that's what's happening here, but once the main thread goes to sleep after starting the workers, the eighth runnable worker thread should surely start, I'd think. Can anyone give insight into the MP semaphore and scheduling internals? Thanks. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0