January 2008 - Darwin-dev - lists.apple.com

Re: Darwin 8.11 and 9.1 Source
by Rosyna 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 26, 2008, at 4:47 PM, William Siegrist wrote: --- Sincerely, Rosyna Keller Technical Support/Carbon troll/Always needs a hug Unsanity: Unsane Tools for Insanely Great People It's either this, or imagining Phil Schiller in a thong. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Is it? Still not CF, CFNetwork, or Objective-C code for 10.5.1. And the dyld code for 10.4.11 is missing (http://www.opensource.apple.com/darwinsource/10.4.11.ppc/dyld-46.16/ ) and it's something I really need.... The CF source is also missing for 10.4.11. The Darwin team is pleased to announce the availability of the source code for Darwin 8.11 and 9.1. These sources correspond to Mac OS X 10.4.11 and 10.5.1 respectively, for both PowerPC- and Intel-based Macintoshes. This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Andrew Gallatin 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com James Peach writes: > On 28/01/2008, Andrew Gallatin <gallatin(a)cs.duke.edu> wrote: > > > > Peter Seebach writes: > > > In message <74ED7582-F08C-4A48-A3FB-B250951A1E62(a)elis.ugent.be>, Jonas Maebe wr > > > ites: > > > >If it only hurt performance a bit I would never have changed the code > > > >from using fork (which is what we used on all other *nix ports) into > > > >using vfork in the first place. But a 25% to 40% slowdown caused by > > > >173 system calls in the process of compiling about 180 kloc is > > > >astronomical in my view. > > > > > > It does seem unusually large, and I'd be interested in seeing what caused it. > > > > There was an interesting fork/exec performance thread quite > > a while back. (http://lists.apple.com/archives/darwin-kernel/2002/Sep/msg00060.html) > > > > I see they've fixed the underlying problem in this particular thread > > (which turned out to be that the fork/exec cost scaled poorly with > > stack size), but shell scripts (expecially configure) remain painfully > > slow compared to Linux or Solaris on the same hardware. Lmbench > > still shows fork+exec taking roughly 5x-10x as long on MacOSX as > > Linux on the same hardware. > > One thing that speeds up configure noticeably for me is to turn off > the Terminal option that updates the window title with the current > process name. I don't use terminal. Or even MacOSX as a desktop. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: posix_spawn(2) (Was: Re: [Fwd: Re: execv bug???])
by Kevin Van Vechten 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 28, 2008, at 12:00 AM, Steve Checkoway wrote: On Jan 27, 2008, at 7:23 PM, Jordan K. Hubbard wrote: - Kevin _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… In another couple of releases, posix_spawn(2) will also hopefully be far enough in our rear view mirrors that developers can standardize on it. I've been curious, what are the advantages to using posix_spawn(2) instead of fork(2) + execve(2)? posix_spawn(2) provides the performance benefit of vfork(2) + execve(2) while still allowing you to manipulate attributes of the child process that could not safely be done with vfork(2). And it's arguably a cleaner API for process creation (though that's more a matter of style). This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by James Peach 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=TLqzQokzAQR9ou7BEq/BUOnbMteq86W9swu6F2rIkKk=; b=NQXJk9Z9Kcooi3NGu12Cidn0Tz5FsR0eag/YX0HO/ur4GZRiN1y1YuiqN3r5hmObt8cGp/tBueBOHqfIeBjAF60lRNnQrThYS/f6xJDfW4UCFxVvKVBeFhh4S8NkcsLh8a0oQ6ChWMexizImvedXkxNgSs1Qc6C9A9A7AhvPjcw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=HULluI0IQGqgXBMqehSujB3GmEHSXDuO3E/DhsVnTkVfB4qvwPaeEunm6nBUkG6q5KtctYw6M0bZS9RGXGwMkhEzJ2rJ/TujhS4tWYrfe/nyA0FEocBQlH/vr+HQNnIhVbSdU4gxrxzr6eNqlErD1YW4DBIMcIm7DeFJdj0HSM4= On 28/01/2008, Andrew Gallatin <gallatin(a)cs.duke.edu> wrote: > > Peter Seebach writes: > > In message <74ED7582-F08C-4A48-A3FB-B250951A1E62(a)elis.ugent.be>, Jonas Maebe wr > > ites: > > >If it only hurt performance a bit I would never have changed the code > > >from using fork (which is what we used on all other *nix ports) into > > >using vfork in the first place. But a 25% to 40% slowdown caused by > > >173 system calls in the process of compiling about 180 kloc is > > >astronomical in my view. > > > > It does seem unusually large, and I'd be interested in seeing what caused it. > > There was an interesting fork/exec performance thread quite > a while back. (http://lists.apple.com/archives/darwin-kernel/2002/Sep/msg00060.html) > > I see they've fixed the underlying problem in this particular thread > (which turned out to be that the fork/exec cost scaled poorly with > stack size), but shell scripts (expecially configure) remain painfully > slow compared to Linux or Solaris on the same hardware. Lmbench > still shows fork+exec taking roughly 5x-10x as long on MacOSX as > Linux on the same hardware. One thing that speeds up configure noticeably for me is to turn off the Terminal option that updates the window title with the current process name. -- James Peach | jorgar(a)gmail.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Andrew Gallatin 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Peter Seebach writes: > In message <74ED7582-F08C-4A48-A3FB-B250951A1E62(a)elis.ugent.be>, Jonas Maebe wr > ites: > >If it only hurt performance a bit I would never have changed the code > >from using fork (which is what we used on all other *nix ports) into > >using vfork in the first place. But a 25% to 40% slowdown caused by > >173 system calls in the process of compiling about 180 kloc is > >astronomical in my view. > > It does seem unusually large, and I'd be interested in seeing what caused it. There was an interesting fork/exec performance thread quite a while back. (http://lists.apple.com/archives/darwin-kernel/2002/Sep/msg00060.html) I see they've fixed the underlying problem in this particular thread (which turned out to be that the fork/exec cost scaled poorly with stack size), but shell scripts (expecially configure) remain painfully slow compared to Linux or Solaris on the same hardware. Lmbench still shows fork+exec taking roughly 5x-10x as long on MacOSX as Linux on the same hardware. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Jonas Maebe 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 28 Jan 2008, at 16:26, Dave Zarzycki wrote: On Jan 28, 2008, at 4:28 AM, Jonas Maebe wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… If it only hurt performance a bit I would never have changed the code from using fork (which is what we used on all other *nix ports) into using vfork in the first place. But a 25% to 40% slowdown caused by 173 system calls in the process of compiling about 180 kloc is astronomical in my view. Well, if performance really mattered, then multiple processes wouldn't be used. Debating fork() versus vfork() is like debating whether you can run faster with 10 or 20 pounds attached to your ankles. Either way, you're going to run slower than the guy with no weights attached to his ankles. I'm calling the (default, system-supplied) assembler from the compiler once per compiled source file. I think that is a perfectly normal use- case and not an example of using the wrong tools/api's for the job (afaik, the assembler does not support batch processing). The people maintaining the Windows/Linux ports have created i386/ x86_64 internal assemblers for elf/(pe)coff (and recently also an internal linker for those targets), but that was mainly because a couple of years back, the only way to achieve dead code stripping with the GNU linker was to put every symbol in a separate object file. And assembling thousands of object files per source file indeed completely kills performance no matter how efficient fork/vfork/posix_spawn/ CreateProcess/... is (well, that and also some general hostility of depending on external tools, but I don't share that sentiment). When I did those tests, the total time spent on calling the assembler and letting it assemble was 3.5 seconds (out of a total of 15 seconds for compiling + assembling + linking) when using vfork on a G5/1.8GHz. I just didn't consider it worth it to spend a lot of implementation and debugging (and future maintenance) time on reducing 3.5 seconds for compiling 180 klocs of code into 2 or 1 second(s). Writing an internal assembler is a non-trivial amount of work to reinvent the wheel, as far as I'm concerned (not to mention that I'm not an ADC Select member and hence only can find out what's been broken after new Xcode releases are out, and enough things break already when using the external assembler/linker due to doing things slightly different than gcc which are hence not caught by Apple's internal testing). In any case, if you really want all sorts of forwards and backwards compatibility, our dynamic linker APIs will let you, dynamically probe the presence of posix_spawn. That will let you use the API, but default back to vfork (or fork or whatever else you like) on older releases. Some dynamic linker api's themselves are also not implemented in older versions (<= 10.2 has no native dl*) and, iirc, others have also changed behaviour (and deprecation status) over time. It's not that simple to add this sort of stuff in a non-cluttering and cross- platform way. I agree that in general, this is however indeed the way to go, but I guess I'm mainly a bit miffed because a) our Linux and FreeBSD ports use only direct syscalls and no libc stuff, because glic breaks backwards binary compatibility every other full moon, while syscalls are an extremely stable interface (we've never had a newer Linux kernel breaking our old programs, and we've supported Linux since 1995 or so) b) when I first asked about that on this list a long time ago (http://lists.apple.com/archives/darwin-development/2003/May/msg00185.html ), I was told that using syscalls on darwin would be an extremely bad idea (which I agree with) c) I concluded from this that that I should libc instead as a system interface, and that it would be just as stable (if not more so) since Apple is a commercial company The fact that even this interface may not remain backward compatible is disappointing however. When I first learned about the concept of frameworks, I (naively) assumed this meant that breaking backwards library compatibility would no longer be required, since you could just bump the compatibility version so all older programs could transparently keep using the older version of the framework. Obviously, for something like libSystem this is not a solution "for free" since you'd still have to keep patching the old version to keep running on the new syscall interfaces if older syscalls are removed/ changed. But it's not a solution for higher level frameworks either if even libSystem isn't safe. So I'm starting to wonder how it is supposed to work at all. This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by seebs＠seebs.net 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com In message <74ED7582-F08C-4A48-A3FB-B250951A1E62(a)elis.ugent.be>, Jonas Maebe wr ites: >If it only hurt performance a bit I would never have changed the code >from using fork (which is what we used on all other *nix ports) into >using vfork in the first place. But a 25% to 40% slowdown caused by >173 system calls in the process of compiling about 180 kloc is >astronomical in my view. It does seem unusually large, and I'd be interested in seeing what caused it. -s _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Dave Zarzycki 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 28, 2008, at 4:28 AM, Jonas Maebe wrote: On 28 Jan 2008, at 02:15, Peter Seebach wrote: Yes. It does make a difference, but even so, the API is deprecated, so even if this hurts performance a bit, it's still probably best to avoid it whenever possible. davez _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… In message <9F31611D-701E-4351-B63D-916E520C584A(a)elis.ugent.be>, Jonas Maebe wr ites: That's more or less true for Linux, but not for Mac OS X at least up till 10.4.x (I haven't benchmarked on 10.5 yet). Compiling our compiler with itself, which involves about 173 (v)fork+execs from a single compiler run to assemble&link all the files, is 20% to 25% slower with fork instead of vfork on a G4, and 35% to 40% on a G5 (32 bit processes in both cases) on 10.4.x. And for clarity: this is relative to the entire time needed for compiling+assembling+linking everything (on the G5: 24 vs 15 seconds), not some academic mbench- like speed difference between the fork and vforks. If it only hurt performance a bit I would never have changed the code from using fork (which is what we used on all other *nix ports) into using vfork in the first place. But a 25% to 40% slowdown caused by 173 system calls in the process of compiling about 180 kloc is astronomical in my view. Well, if performance really mattered, then multiple processes wouldn't be used. Debating fork() versus vfork() is like debating whether you can run faster with 10 or 20 pounds attached to your ankles. Either way, you're going to run slower than the guy with no weights attached to his ankles. In any case, if you really want all sorts of forwards and backwards compatibility, our dynamic linker APIs will let you, dynamically probe the presence of posix_spawn. That will let you use the API, but default back to vfork (or fork or whatever else you like) on older releases. This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Jonas Maebe 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 28 Jan 2008, at 02:15, Peter Seebach wrote: That's more or less true for Linux, but not for Mac OS X at least up till 10.4.x (I haven't benchmarked on 10.5 yet). Compiling our compiler with itself, which involves about 173 (v)fork+execs from a single compiler run to assemble&link all the files, is 20% to 25% slower with fork instead of vfork on a G4, and 35% to 40% on a G5 (32 bit processes in both cases) on 10.4.x. And for clarity: this is relative to the entire time needed for compiling+assembling+linking everything (on the G5: 24 vs 15 seconds), not some academic mbench- like speed difference between the fork and vforks. Yes. It does make a difference, but even so, the API is deprecated, so even if this hurts performance a bit, it's still probably best to avoid it whenever possible. Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… In message <9F31611D-701E-4351-B63D-916E520C584A(a)elis.ugent.be>, Jonas Maebe wr ites: If it only hurt performance a bit I would never have changed the code from using fork (which is what we used on all other *nix ports) into using vfork in the first place. But a 25% to 40% slowdown caused by 173 system calls in the process of compiling about 180 kloc is astronomical in my view. This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Jonas Maebe 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 28 Jan 2008, at 04:23, Jordan K. Hubbard wrote: n Jan 27, 2008, at 2:46 PM, Jonas Maebe wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… That's more or less true for Linux, but not for Mac OS X at least up till 10.4.x (I haven't benchmarked on 10.5 yet). Compiling our compiler with itself, which involves about 173 (v)fork+execs from a single compiler run to assemble&link all the files, is 20% to 25% slower with fork instead of vfork on a G4, and 35% to 40% on a G5 (32 bit processes in both cases) on 10.4.x. And for clarity: this is relative to the entire time needed for compiling+assembling +linking everything (on the G5: 24 vs 15 seconds), not some academic mbench-like speed difference between the fork and vforks. It would be interesting to benchmark this in 10.5 as well, given a number of changes to the relevant code, I don't have a PPC with 10.5 yet, but I'll try when I have the chance. I'll also check fork vs vfork on my MacBook (which can only run 10.5). but the original poster who noted that vfork() was deprecated was correct. At some point, I would not be at all surprised to see a "#define vfork fork" in unistd.h so that code path can be deprecated entirely. What about backwards compatibility for already compiled binaries? Since libSystem is not a framework, I'm not sure how this can be ascertained should vfork also be removed from the library itself. This email sent to site_archiver(a)lists.apple.com

1 0