January 2008 - Darwin-dev - lists.apple.com

Re: Darwin 8.11 and 9.1 Source
by David M. Cotter 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Yes, the new CFLite is what I really want, when will that be available? -dave _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Probably not the Darwin team's fault, but the Obj-C and CoreFoundation sources still aren't up. :-( This email sent to site_archiver(a)lists.apple.com

1 0

Re: Darwin 8.11 and 9.1 Source
by Joe Ranieri 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 26, 2008, at 18:47, William Siegrist wrote: Joe Ranieri Lead Magician, Alacatia Labs, Inc. http://alacatia.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The Darwin team is pleased to announce the availability of the source code for Darwin 8.11 and 9.1. These sources correspond to Mac OS X 10.4.11 and 10.5.1 respectively, for both PowerPC- and Intel-based Macintoshes. The sources are available at: Probably not the Darwin team's fault, but the Obj-C and CoreFoundation sources still aren't up. :-( This email sent to site_archiver(a)lists.apple.com

1 0

Darwin 8.11 and 9.1 Source
by William Siegrist 26 Jan '08

26 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com http://www.opensource.apple.com/darwinsource/10.4.11.ppc/ http://www.opensource.apple.com/darwinsource/10.4.11.x86/ http://www.opensource.apple.com/darwinsource/10.5.1/ ---- William Siegrist wsiegrist(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com The Darwin team is pleased to announce the availability of the source code for Darwin 8.11 and 9.1. These sources correspond to Mac OS X 10.4.11 and 10.5.1 respectively, for both PowerPC- and Intel-based Macintoshes. The sources are available at: smime.p7s

1 0

Re: perl 5.8.8, backtick execution and leopard
by Dave Zarzycki 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything I should know about thread safety and fprintf?
by Jonathan del Strother 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=/UN5F2Sr3C65s7m1Bn9kaQHfQPnu7yiuOMVn89JrFXE=; b=fowA6FbUGLsRQi9cOVdFt42W6SsPXx54kVnZTXOTtV3ndMDD6JxOwseSEA7uDDDh46hfzdFzj+vobShr8TSw+BrvanUqncdu4mkHHI5g6iyziE0xe3vpnuBfrl+cu/WchoLwXD0F+SlVbTgfX/yBRrdsz/gAtGly/7H+TVcuBdw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=UvrWNLPT4d2p8XD/nXrwxd7WMhEx2fEfFMOBDDOjj9PtHXNulPXj+rWaG42HAn5G3QbtKnxx471trmy5UGpdkfUGZpKxa3R+Xn8cINLWpPK9l07VO4F9aQ7L48j5qH2vp5OnKr81urGhzHpJ0g961Fp74rApvN2AohqXgV8d9w4= On Jan 24, 2008 1:42 PM, Jonathan del Strother <maillist(a)steelskies.com> wrote: > On Jan 24, 2008 1:12 PM, Terry Lambert <tlambert(a)apple.com> wrote: > > Please see my other posting. You are not fixing your underlying > > problem here. > > > > My personal suggested fix would be to marshal the data to a single > > thread for disposition. Then your limit will be however much data you > > can jam down the drain pipe plus however much data you are willing to > > internally queue in your application in the marshalling process. > > > > I'm only logging a few lines - maybe 500 characters in total - so I > can't see how I'd be overloading the queue. Even if I did, presumably > the write() call ought to return eventually? > > I haven't yet been able to reproduce this outside of Xcode - it's > tempting to blame it on an oddity in Xcode's run log and forget about > it... *Finally* found the problem, and am feeling dumb for not spotting the pattern before. Obviously it was nothing to do with fprintf and/or threads. In certain conditions, my application relaunches itself. When it does so, the new process would manage a few log messages, then the original process would quit, and the new process would block the next time it tried to write to stderr. So, I assume that the file handles created for the original process are getting inherited through to the new process, but not surviving once the old process exits. Looks like I need to find a better way of relaunching... Thanks for all the help and suggestions, Jon _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything I should know about thread safety and fprintf?
by Jonathan del Strother 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=Hr8mF20Zo4G7/bGVkdqUxu8jWKnesk2sl98VfXRGzhM=; b=j0QDaZyipqZW5WOCcT3csJxc1TGm3IX4LUw4UW1ezd0c3/kMJPpdPCRm5sAOMve3z/IaevWV5xtUNjk07RxRCt8X1Yl3iBU2vvCnXJQ55MWqkDvYcL3ToHVkife3t65iFp++qADCOnJ3utO/1Veql4m4qtShgmleIFJk18SSdu4= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=htLhCvuJsV7K/5DS5BrO9VTNwX9CAqb9BoAvY+Wn5f01sIFLCvsZdYNFhrFc5UWhONwzwa8eBHXxgU7wYc0Qdy+0YIZBh6ij46SOQSlXmC1NDb2dOFzGmvOmgspXVmtlfsoNYOBQwEujC3prVPvjIXWY6ccMxK93q0RAhl6ffqw= On Jan 24, 2008 1:12 PM, Terry Lambert <tlambert(a)apple.com> wrote: > Please see my other posting. You are not fixing your underlying > problem here. > > My personal suggested fix would be to marshal the data to a single > thread for disposition. Then your limit will be however much data you > can jam down the drain pipe plus however much data you are willing to > internally queue in your application in the marshalling process. > I'm only logging a few lines - maybe 500 characters in total - so I can't see how I'd be overloading the queue. Even if I did, presumably the write() call ought to return eventually? I haven't yet been able to reproduce this outside of Xcode - it's tempting to blame it on an oddity in Xcode's run log and forget about it... _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything I should know about thread safety and fprintf?
by Terry Lambert 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry On Jan 23, 2008, at 5:02 PM, Terry Lambert wrote: The only real way to talk to the system logging facility is via syslog() Damn, spoke too soon. Now my app gets stuck in #0 0x94709a31 in write$NOCANCEL$UNIX2003 () #1 0x94700714 in asl_send () #2 0x946ff501 in asl_vlog () #3 0x946ff267 in asl_log () With a call to asl_log looking something like : asl_log(aslClient, NULL, ASL_LEVEL_INFO, "%s:%d - %s", "MillicentView.m", 236, "qc - 2.000000, 1.500000"); I attached gdb, and tried to 'finish' out of the write() frame, which never returns. The arguments look a little odd - presumably the function definition is along the lines of write(int fd, const void *buffer, size_t numbytes) ? I get : (gdb) p (char*) *(int *)($ebp+12) // buffer argument $1 = 0x15778380 "" (gdb) p *(size_t *)($ebp+16) // numbytes argument $2 = 360399840 I'm going to carry on hacking away at this, see if I can reduce the problem some more. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Please see my other posting. You are not fixing your underlying problem here. My personal suggested fix would be to marshal the data to a single thread for disposition. Then your limit will be however much data you can jam down the drain pipe plus however much data you are willing to internally queue in your application in the marshalling process. If your data is bursty, then a large enough queue retention pool for the internal queue will let things work. Think of it as pouring water in the top of a bucket with a small hole in the bottom; as long as you only pour in as much water on average as can drain out the hole, then burstiness is about how big you need to make the bucket so it never overflows (in queueing theory, this is the instantanious maxima for pool retention time). NB: If you are doing this to log e.g. kernel events that are stalled pending user space operations, you had better be _certain_ to exempt everything in the code path that gets things logged from logging or the need to go to user space on the first place. On Jan 24, 2008, at 4:51 AM, Jonathan del Strother <maillist(a)steelskies.com > wrote: On Jan 24, 2008 11:16 AM, Jonathan del Strother <maillist(a)steelskies.com > wrote: On Jan 24, 2008 7:00 AM, Jordan K. Hubbard <jkh(a)apple.com> wrote: Which is how you should generate log messages only if you wish to use the deprecated, purely-for-backwards-compatibility interface. :-) Please use asl(3) for all new code. It's a far better, more flexible interface. Ah. I thought the standard way of system logging was just fprintf(stderr, ... ). I still have no idea why I should be getting stuck in write() like that, but I've switched to asl and the problem seems to have magically disappeared... ... which seems an unusually large buffer size, assuming I'm reading that right. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything I should know about thread safety and fprintf?
by Terry Lambert 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Ah. I thought the standard way of system logging was just fprintf(stderr, ... ). -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I don't believe there is anything special about stdout and stdin. They're just FILE* that initially correspond to file descriptors 1 and 2. They can be changed with freopen(3). OK, I guess some more clarification is in order, since the problem was "fixed", but not "really fixed"... They (and stderr) are special in threaded programs in that threads libraries implementing stdio must serialize access to FILE *'s, the contents of which are process scope global data, not thread scoped data. What the get connected to depends on what the parent process connects the fds to, or, if freopen() or some other call moves the (another example would be explicitly closing and opening the underlying fds, which are stored as dumb integers in the FILE *, which has no concept of what it is actually talking to). If you implement your own operations on FILE *'s, to be thread safe, you have to also participate in the serialization protocol by using filelock(3) (it even has a man page, and it's visible in the problem thread stack). Since buffered I/O buffering happens in the stdio library in user space, the only way to do this is to either disable buffering (see below, though), which can have bad side effects for some underlying file types, or hold the lock describrd above over any operation capable of changing buffer descriptor contents or metadata (i.e. the FILE *). When this happens, blocking on output blocks everyone else attempting to take the lock on the same FILE * behind acquiring the lock. In normal pthreads programs, the programmer will usually avoid the use of FILE * altogether to avoid the serialization point (e.g. because the reason they were using threads in the first place was increased concurrency), OR call setbuf() to disable buffering (and hope/pray that the stdio library author took setbuf() into account, and made the microoptimization of avoiding locking in this one case: usually a vain hope), OR use stdio from a single thread (marshalling all data output to that thread to avoid contention for the FILE * lock). The third one gets a small performance win, if the output data tends to be, on average, half or less of the default (or selected) stdio buffer size, since it avoids I/O at the cost of acquiring and releasing an uncontested lock on the buffer. The smaller the writes and the longer the time before an automatic or forced flush, the bigger the win, of course. In general, it's more or less impossible to usefully use the same FILE * from multiple threads in a single process, if the data sink is unable to keep up with the output data rate, or is of a type which can have I/O stalls at all. Even if you don't stall (e.g. you are writing to a local disk drive that is spun ip and responding to all I/O requests), the serialization overhead can result in up to a 2X slowdown. This is due to holding the FILE * lock over both the copy from the data source (and any formatting processing needed), and the flush of the FILE * buffer contents to the real underlying fd (copying from user to kernel space). Note that ASL and syslog both have this same serialization for formatted data, and because both use sockets on a local connection, you get a transmit queue depth and a receive queue depth of 64K before you see a stall (pipes are 4K, remote fs's and sockets are local send queue + remote receive queue + TCP window size). A tty/pty is ~4K (this is why the paste limit in terminal windows, if you use a shell that's not smart enough to rebuffed into its internal command line buffer). Other things that can be represented by fds and which provide reliable stream deliver have their own limits. The reason the original poster doesn't see the problem with syslog/ASL at this point is only because the other end is keeping up with the load (for now). By the way: the multiple threads accessing the same serialized API problem is a design problem, and hasn't gone away, it's just been submerged until the load/thread count goes up high enough in the program doing the logging, or until a high enough priority process loads the system to the point it starts robbing cycles from the logging daemon on the other side of the connection, and it can no longer keep up. At that point, the write will stall again. If this "can't happen" because more threads will never be created, and the system is dedicated to a single task, so it doesn't need to worry about competitive loading, then this warning can probably be safely ignored, as long as there's never any future changes to the code or system load invalidate these conditions. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything I should know about thread safety and fprintf?
by Jonathan del Strother 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=3W6TdG9CzIpuLkM2+MMokR7n4wxCWoFiflvl3bsgVJA=; b=n5Ww0DnuCiQp7MWE8uNCebVc6G6mlMAg/rAGTUA9DXPndJ81l/0GYJ7W6DdGtSXph8qsmB/Es98hnyUovE7TeLOdscgg8sc1lX+2SHLIWTYakIxWK6ClUYN68QqxCI8Vt3ENY4vptSOLTnY+JG6ndjev3IsjgvbHK2iSXQlkQOw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=c8pyWagvUHqPCI8hI7rhCrXrCHHCV1r3byQ+iN32gzFKzY9DpDF4eShdMGoQ2DDaF8aDdoZf1OkyIZ0rLZnHKRu70k2RHFFd7O+EUsznbnciS7W+gAWjphI1aoVs3UL74hJcYZuAy47PB1lt6b6q7xIiwfIgrxEW8HLuM9Pm+/A= On Jan 24, 2008 11:16 AM, Jonathan del Strother <maillist(a)steelskies.com> wrote: > > On Jan 24, 2008 7:00 AM, Jordan K. Hubbard <jkh(a)apple.com> wrote: > > > > On Jan 23, 2008, at 5:02 PM, Terry Lambert wrote: > > > > > The only real way to talk to the system logging facility is via > > > syslog() > > > > Which is how you should generate log messages only if you wish to use > > the deprecated, purely-for-backwards-compatibility interface. :-) > > > > Please use asl(3) for all new code. It's a far better, more flexible > > interface. > > > Ah. I thought the standard way of system logging was just > fprintf(stderr, ... ). > I still have no idea why I should be getting stuck in write() like > that, but I've switched to asl and the problem seems to have magically > disappeared... Damn, spoke too soon. Now my app gets stuck in #0 0x94709a31 in write$NOCANCEL$UNIX2003 () #1 0x94700714 in asl_send () #2 0x946ff501 in asl_vlog () #3 0x946ff267 in asl_log () With a call to asl_log looking something like : asl_log(aslClient, NULL, ASL_LEVEL_INFO, "%s:%d - %s", "MillicentView.m", 236, "qc - 2.000000, 1.500000"); I attached gdb, and tried to 'finish' out of the write() frame, which never returns. The arguments look a little odd - presumably the function definition is along the lines of write(int fd, const void *buffer, size_t numbytes) ? I get : (gdb) p (char*) *(int *)($ebp+12) // buffer argument $1 = 0x15778380 "" (gdb) p *(size_t *)($ebp+16) // numbytes argument $2 = 360399840 ... which seems an unusually large buffer size, assuming I'm reading that right. I'm going to carry on hacking away at this, see if I can reduce the problem some more. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything I should know about thread safety and fprintf?
by Steve Checkoway 24 Jan '08

24 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 24, 2008, at 3:16 AM, Jonathan del Strother wrote: Ah. I thought the standard way of system logging was just fprintf(stderr, ... ). -- Steve Checkoway "Anyone who says that the solution is to educate the users hasn't ever met an actual user." -- Bruce Schneier _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com I don't believe there is anything special about stdout and stdin. They're just FILE* that initially correspond to file descriptors 1 and 2. They can be changed with freopen(3). smime.p7s

1 0