January 2008 - Darwin-dev - lists.apple.com

Re: posix_spawn(2) (Was: Re: [Fwd: Re: execv bug???])
by Clark Cox 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=G5y2oGixw3P9njTG33vpb6RbfVmTb54KpDEtL1EqbZ8=; b=pSw5Btd54gExOF4PvlblAuIp5K2QEVZcac3mKQfobZTr+20dz8G1jxI7DCwmYE0xbHy+jqwA9yF5t95pDuLG8UsAAVP0xQNv75rAU8JmRkdF+Xdfso3tcbtgK1Bvamk5hNOdC/9AUcYN7HgkYin//OF3dJTDraMAhDutmSR0N9Q= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=K3FJUYFWmQ0WWVcZKpsrsE8b8ILtki6vUnLXvDVtYSzIHksNMjB6cMkCvq+eB7Lr5xk0xPxkYFhOq3QRovjercGNYPIMCms6KHQHuIP/YFeL92n3h8De886+x0tOgll14fowRkPSaqkmc9IXmlHjPu6kjRQDdK+GpS4Egw1GHLs= On 1/28/08, Steve Checkoway <s(a)pahtak.org> wrote: > > On Jan 27, 2008, at 7:23 PM, Jordan K. Hubbard wrote: > > > In another couple of releases, posix_spawn(2) will also hopefully > > be far enough in our rear view mirrors that developers can > > standardize on it. > > > I've been curious, what are the advantages to using posix_spawn(2) > instead of fork(2) + execve(2)? One main benefit, as I see it, is that posix_spawn avoids the address space duplication implied by fork, which can be quite a time-waster when all you're going to do is blow that space away with an exec. -- Clark S. Cox III clarkcox3(a)gmail.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

posix_spawn(2) (Was: Re: [Fwd: Re: execv bug???])
by Steve Checkoway 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 27, 2008, at 7:23 PM, Jordan K. Hubbard wrote: -- Steve Checkoway "Anyone who says that the solution is to educate the users hasn't ever met an actual user." -- Bruce Schneier _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com In another couple of releases, posix_spawn(2) will also hopefully be far enough in our rear view mirrors that developers can standardize on it. I've been curious, what are the advantages to using posix_spawn(2) instead of fork(2) + execve(2)? smime.p7s

1 0

Re: [Fwd: Re: execv bug???]
by Jordan K. Hubbard 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 27, 2008, at 2:46 PM, Jonas Maebe wrote: - Jordan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… That's more or less true for Linux, but not for Mac OS X at least up till 10.4.x (I haven't benchmarked on 10.5 yet). Compiling our compiler with itself, which involves about 173 (v)fork+execs from a single compiler run to assemble&link all the files, is 20% to 25% slower with fork instead of vfork on a G4, and 35% to 40% on a G5 (32 bit processes in both cases) on 10.4.x. And for clarity: this is relative to the entire time needed for compiling+assembling+linking everything (on the G5: 24 vs 15 seconds), not some academic mbench- like speed difference between the fork and vforks. It would be interesting to benchmark this in 10.5 as well, given a number of changes to the relevant code, but the original poster who noted that vfork() was deprecated was correct. At some point, I would not be at all surprised to see a "#define vfork fork" in unistd.h so that code path can be deprecated entirely. In another couple of releases, posix_spawn(2) will also hopefully be far enough in our rear view mirrors that developers can standardize on it. This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by seebs＠seebs.net 28 Jan '08

28 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com In message <9F31611D-701E-4351-B63D-916E520C584A(a)elis.ugent.be>, Jonas Maebe wr ites: >That's more or less true for Linux, but not for Mac OS X at least up >till 10.4.x (I haven't benchmarked on 10.5 yet). Compiling our >compiler with itself, which involves about 173 (v)fork+execs from a >single compiler run to assemble&link all the files, is 20% to 25% >slower with fork instead of vfork on a G4, and 35% to 40% on a G5 (32 >bit processes in both cases) on 10.4.x. And for clarity: this is >relative to the entire time needed for compiling+assembling+linking >everything (on the G5: 24 vs 15 seconds), not some academic mbench- >like speed difference between the fork and vforks. Yes. It does make a difference, but even so, the API is deprecated, so even if this hurts performance a bit, it's still probably best to avoid it whenever possible. >The modern alternative is obviously posix_spawn, but we cannot use >that since we still support 10.3.x (it even probably even still works >on 10.1 and 10.2 as well, but I don't test those anymore). True. That said, the code in question is a bug according to the documented semantics of vfork on OS X. -s _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by Jonas Maebe 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 27 Jan 2008, at 20:56, Peter Seebach wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… In message <479CD1DB.4030504(a)yahoo.com>, Steve writes: So, are you suggesting then it could well be a bug with Leopard (it works fine on Tiger and, it works fine as coded by simply removing the v from vfork)? Obviously, the simple solution on Leopard for now is to use fork. I shall proceed to make a small test program and submit to Apple then. I wouldn't consider it a significant bug; vfork is deprecated, having been a performance hack to improve performance on the VAX. The existence of modern systems with copy-on-write semantics makes it a waste of time to try to make things work with vfork That's more or less true for Linux, but not for Mac OS X at least up till 10.4.x (I haven't benchmarked on 10.5 yet). Compiling our compiler with itself, which involves about 173 (v)fork+execs from a single compiler run to assemble&link all the files, is 20% to 25% slower with fork instead of vfork on a G4, and 35% to 40% on a G5 (32 bit processes in both cases) on 10.4.x. And for clarity: this is relative to the entire time needed for compiling+assembling+linking everything (on the G5: 24 vs 15 seconds), not some academic mbench- like speed difference between the fork and vforks. Copying the memory maps is extremely expensive on Darwin (at least in the versions I tested on), and gets progressively more expensive the more memory the program has allocated. The modern alternative is obviously posix_spawn, but we cannot use that since we still support 10.3.x (it even probably even still works on 10.1 and 10.2 as well, but I don't test those anymore). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Running a time consuming background process
by Stephen Hoffman 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Organization: HoffmanLabs LLC User-agent: Thunderbird 2.0.0.9 (Macintosh/20071031) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I need to run a custom program that I expect to need several weeks of (wall clock) time to complete. How can I set up and run so that I can save state on any signal, and having my program automatically resumed on start up, with as little extra work as possibly when resuming. My program is a pure batch job, by the way. Why not checkpoint your progress and your run-time state on a periodic basis? This approach is simple, easier to verify, and catches a hard power-down and most anything else. In my experience, there are cases where catching errors can be too clever by half. What happens if you catch an error that means you can't, for instance, write? How and what to checkpoint varies greatly, based on what you are doing. Transactional processing and databases work well here for a subset of the possible cases. This is less efficient (proportional to the quantity of data and the frequency of the checkpoints), but this approach is also easier to code, test and maintain. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Running a time consuming background process
by Jordan K. Hubbard 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com - Jordan On Jan 27, 2008, at 10:56 AM, Tommy Nordgren wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/jkh%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Mmmm. Process snapshotting. I'm afraid that feature does not exist on Mac OS X, so you'll have to simulate this yourself with manual labor behind the scenes. Sweep all of your program's state into well- known data structures, save those data structures when you get the signal, add logic to the program which detects saved state files and allows itself to resume from them, you're done. It's all "just a simple matter of coding" as we like to say. :-) I need to run a custom program that I expect to need several weeks of (wall clock) time to complete. How can I set up and run so that I can save state on any signal, and having my program automatically resumed on start up, with as little extra work as possibly when resuming. My program is a pure batch job, by the way. ---------------------------------- Skinheads are so tired of immigration, that they are going to move to a country that don't accept immigrants! Tommy Nordgren tommy.nordgren(a)comhem.se This email sent to jkh(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: [Fwd: Re: execv bug???]
by seebs＠seebs.net 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com In message <479CD1DB.4030504(a)yahoo.com>, Steve writes: >So, are you suggesting then it could well be a bug with Leopard (it >works fine on Tiger and, it works fine as coded by simply removing the >v from vfork)? Obviously, the simple solution on Leopard for now is to >use fork. I shall proceed to make a small test program and submit to >Apple then. I wouldn't consider it a significant bug; vfork is deprecated, having been a performance hack to improve performance on the VAX. The existence of modern systems with copy-on-write semantics makes it a waste of time to try to make things work with vfork; further, I wouldn't necessarily expect either process to be stable after calling vfork and then doing anything other than exec* and exit -- there's nothing I know of stating that chdir() calls can't have side-effects that include modification of the caller's address space in some way, and you should never call anything that modifies the caller's address space from the child side of a vfork. See the vfork man page: [EINVAL] A system call other than _exit() or execve() (or libc functions that make no system calls other than those) is called following calling a vfork() call. chdir() is a syscall. It is an error to call chdir() inside the child of a vfork. -s _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Running a time consuming background process
by Tommy Nordgren 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I need to run a custom program that I expect to need several weeks of (wall clock) time to complete. How can I set up and run so that I can save state on any signal, and having my program automatically resumed on start up, with as little extra work as possibly when resuming. My program is a pure batch job, by the way. ---------------------------------- Skinheads are so tired of immigration, that they are going to move to a country that don't accept immigrants! Tommy Nordgren tommy.nordgren(a)comhem.se This email sent to site_archiver(a)lists.apple.com

1 0

[Fwd: Re: execv bug???]
by Steve 27 Jan '08

27 Jan '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:Content-Type:Content-Transfer-Encoding; b=vz2/PNXtzWkQydvhD6PxzsH02XsvrfUo2u9NDCkFOfwTjVqzIF4F9jWA4UcaWLaByEWLZWqhBFxXxemcAgpFoo1fjXuuzKl4aWFWXjVY2r10FZeI4Lh0HZLdnE0Jjx2uxhFAWH0QzAZCpUc73STv22lWiuo2G5wHcANvbEv7szI= ; User-agent: Thunderbird 2.0.0.9 (Macintosh/20071031) In message 479C488E.7020605(a)yahoo.com"><479C488E.7020605(a)yahoo.com>, Steve writes: Could this actually be a combination of execv AND the use of vfork? Yes. So, is this a known issue perhaps? Sounds likely! _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0