July 2007 - Darwin-dev - lists.apple.com

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Shawn Erickson 10 Jul '07

10 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jul 9, 2007, at 10:02 PM, Mike Frysinger wrote: On Jul 9, 2007, at 6:03 PM, Mike Frysinger wrote: >> >> <http://developer.apple.com/bugreporter/> >> > >> > thanks, ive posted it there as #5322203 >> >> There's already a bug filed on this, and it's currently "Not to be >> fixed". > > can you post the URI to said report please Not really; it wouldn't be visible externally anyway. It's specific to an explicit call to ftruncate(), but it's the same root cause in both instances. -Shawn _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 7/9/07, Terry Lambert <tlambert(a)apple.com> wrote: i keep forgetting apple's definition of "open" != "open" Apple never stated that their bug system was open. Like most companies they keep it private... and I thank them for that because I have a few defects filed that reveal information about a product I am working on. This email sent to site_archiver(a)lists.apple.com

1 0

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Mike Frysinger 10 Jul '07

10 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Md/BESZnSVLEjSuH61u6qROAle7hptHLfAkVkWCmbyFOZ5uZ0PIXlfr0F3TqPWZzfv6VRvEygQKUgEQqCxo4IOPW4XK2yaTS2W0AS3DoNZ/R7t3JY6f2cOseFoi7+qG+G08gFOu/SvB0UdEQ5CfhbnTZQzpVpH/tw7knKwiLscc= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=SqoxTqICh+UqnUgpGq486ndnW4xzSRoM/BZFuEUnwGWkw2s+qPIBPYAkZq4muh9IYQNN8oGxfHdaQB7r9r7vLanikUNAaHb4uicjNkIBnuTpIfkLQJPi7n4Ny2eysp4x94IegAVhekZDOQiNsi5pohpfSaatMnVP0pW6rawDNlc= On Jul 9, 2007, at 6:03 PM, Mike Frysinger wrote: >> >> <http://developer.apple.com/bugreporter/> >> > >> > thanks, ive posted it there as #5322203 >> >> There's already a bug filed on this, and it's currently "Not to be >> fixed". > > can you post the URI to said report please Not really; it wouldn't be visible externally anyway. It's specific to an explicit call to ftruncate(), but it's the same root cause in both instances. i keep forgetting apple's definition of "open" != "open" >> Specifically, the modification time will only be changed if the file >> is modified, which it is not, in this case (the POSIX mandate in the >> case of an actual truncation -- "successful completion" -- is "shall >> mark for update", rather than "shall update", in any event, so any >> code with this assumption built into it is inherently racy). > > the backend details of how truncation is handled is irrelevant to > POSIX so long as the behavior dictated is maintained. > > "successful completion" is not defined as "file was actually > truncated" but rather "open() returned a valid file descriptor (and > not an error)". even if this is how POSIX described it, the behavior > of updating *ctime* and not *mtime* doesnt make any real logical > sense. It does; the metadata (file length) was not changed, but the file was opened for write. No metadata change, no mtime change. again, the backend details are irrelevant ... POSIX states: If O_TRUNC is set and the file did previously exist, upon successful completion, open() shall mark for update the st_ctime and st_mtime fields of the file. * the file previously existed * the call to open() returned a valid file descriptor ("successful completion") * the spec makes no concession like "if the file is 0 bytes already, you may skip st_mtime updates" > as for the race condition, if you want to interpret "mark for update" > to mean the times get updated on the fd but the file itself is delayed > for updates, then simply changing to fstat() rather than stat() would > address that concern. Actually, no it would not. Being marked for update is not the same thing as actually having stat information updated. What happens is that you flag the vnode to indicate that the time in the inode/cnode should be set opportunistically. There can be significant latency between setting the "update my ctime" flag, and the ctime actually being updated. again, my view is from POSIX and nothing else. backend details (libc/kernel/whatever) have no bearing on the discussion. I don't really agree with you interpretation of the standard, though, and neither do The Open Group POSIX conformance test suites, since they don't test for this behaviour. >> The assumptions in the Linux kernel build code are wrong; it should >> be >> using "touch", not "open with O_TRUNC". > > the assumption in the kernel build code is that the system is POSIX > compliant basic logic failure: a test case merely proves that everything tested for passes, not that everything possible passes >> See also the Single UNIX Specification: >> >> <http://www.opengroup.org/onlinepubs/009695399/functions/open.html >> > > > that is the spec ive been quoting all along Maybe you could get them to add it as a test case for the test suite, if they (specifically, The Austin Group, actually) agreed with your interpretation of the standard. i'll start a thread there It'd be trivial to call futimes(fd, NULL); after opening the fd, to not depend on the mtime/O_TRUNC interaction assumprion. As a workaround to actually fixing the assumption about mtime in the Linux kernel build system, you could either make the change described above in your own Darwin kernel sources and build yourself a new kernel, or you could use UFS instead of HFS (the easiest way would be to use it in a disk image, to avoid reformatting), since UFS acts this way. my intentions here are not for myself ... i rarely use OS X let alone ever dig into the Darwin kernel or libc nor do i ever plan on doing such a thing -mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 7/9/07, Terry Lambert <tlambert(a)apple.com> wrote: <snip> <http://www.opengroup.org/openbrand/register/brand3555.htm> This email sent to site_archiver(a)lists.apple.com

1 0

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Terry Lambert 10 Jul '07

10 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com >> <http://developer.apple.com/bugreporter/> > > thanks, ive posted it there as #5322203 There's already a bug filed on this, and it's currently "Not to be fixed". Specifically, the modification time will only be changed if the file is modified, which it is not, in this case (the POSIX mandate in the case of an actual truncation -- "successful completion" -- is "shall mark for update", rather than "shall update", in any event, so any code with this assumption built into it is inherently racy). the backend details of how truncation is handled is irrelevant to POSIX so long as the behavior dictated is maintained. "successful completion" is not defined as "file was actually truncated" but rather "open() returned a valid file descriptor (and not an error)". even if this is how POSIX described it, the behavior of updating *ctime* and not *mtime* doesnt make any real logical sense. as for the race condition, if you want to interpret "mark for update" to mean the times get updated on the fd but the file itself is delayed for updates, then simply changing to fstat() rather than stat() would address that concern. Specifically, he's asking for: cp->c_touch_modtime = TRUE; <http://www.opengroup.org/openbrand/register/brand3555.htm> See also the Single UNIX Specification: -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Jul 9, 2007, at 6:03 PM, Mike Frysinger wrote: can you post the URI to said report please Not really; it wouldn't be visible externally anyway. It's specific to an explicit call to ftruncate(), but it's the same root cause in both instances. It does; the metadata (file length) was not changed, but the file was opened for write. No metadata change, no mtime change. Actually, no it would not. Being marked for update is not the same thing as actually having stat information updated. What happens is that you flag the vnode to indicate that the time in the inode/cnode should be set opportunistically. There can be significant latency between setting the "update my ctime" flag, and the ctime actually being updated. prior to the call to hfs_update at the end of the function do_hfs_truncate() in bsd/hfs/hfs_readwrite.c; that just sets a flag. The actual time isn't really updated until some time later. So the time stamp that gets put on the file would depend on when an updating operation too place subsequent to the flag getting set (it being marked for update). I don't really agree with you interpretation of the standard, though, and neither do The Open Group POSIX conformance test suites, since they don't test for this behaviour. The assumptions in the Linux kernel build code are wrong; it should be using "touch", not "open with O_TRUNC". the assumption in the kernel build code is that the system is POSIX compliant <http://www.opengroup.org/onlinepubs/009695399/functions/open.html > that is the spec ive been quoting all along Maybe you could get them to add it as a test case for the test suite, if they (specifically, The Austin Group, actually) agreed with your interpretation of the standard. It'd be trivial to call futimes(fd, NULL); after opening the fd, to not depend on the mtime/O_TRUNC interaction assumprion. As a workaround to actually fixing the assumption about mtime in the Linux kernel build system, you could either make the change described above in your own Darwin kernel sources and build yourself a new kernel, or you could use UFS instead of HFS (the easiest way would be to use it in a disk image, to avoid reformatting), since UFS acts this way. This email sent to site_archiver(a)lists.apple.com

1 0

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Mike Frysinger 10 Jul '07

10 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=OScnyEwIbMoZkV48eKXQPegspUROi8JRZK3ZCR9q5sfJjBn2xqm1klRe7RDhfSXP03gEmwgViP0GA8CuBTXTqL1XLPc7OoWJ0kjsoXUzNT0/nfJbe4DqSCOG1pnJf57v1fpzDCn3f8cLey3K1DJ3zD8LB/fab1Tjeia8iJww2lI= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=SX+FNHC2yP31gVcvt0Xb8m0VwCbYEELmTAluhSvXBh/7Ca4XS6H9Oim3QyVRHSJrZ3U1P6yXBY0TTi9xMNSXtPu7gHx98yy/LRcM/LUIqUXb4SebTrZEKZLKdkSOtIEhQpDQ6jNYGY4Z7tXTPLjTJTRaWqLg6Us2VWK6+11IrYU= On Jul 9, 2007, at 12:09 PM, Mike Frysinger wrote: > On 7/9/07, Shawn Erickson <shawnce(a)gmail.com> wrote: >> On 7/9/07, Mike Frysinger <vapier.adi(a)gmail.com> wrote: >> > On 6/12/07, Mike Frysinger <vapier.adi(a)gmail.com> wrote: >> > > i was tracking down a bug in the Linux kernel build system >> which seems >> > > to be caused by a bug in Darwin. the kernel build system >> creates a >> > > bunch of 0 byte files and then uses open(O_TRUNC) on them in >> order to >> > > update the st_mtime field (which triggers make to recreate some >> object >> > > files). on my mac mini, this all falls apart. >> > >> > is there somewhere i can file a bug report about this ? i'm really >> > not familiar with Darwin development methodologies >> >> <http://developer.apple.com/bugreporter/> > > thanks, ive posted it there as #5322203 There's already a bug filed on this, and it's currently "Not to be fixed". can you post the URI to said report please It's perfectly legitimate under POSIX to check the length of the file before attempting the truncation, and if the truncation would have no effect, not call into the VOP to do the truncation, thus not modifying the file. Specifically, the modification time will only be changed if the file is modified, which it is not, in this case (the POSIX mandate in the case of an actual truncation -- "successful completion" -- is "shall mark for update", rather than "shall update", in any event, so any code with this assumption built into it is inherently racy). the backend details of how truncation is handled is irrelevant to POSIX so long as the behavior dictated is maintained. "successful completion" is not defined as "file was actually truncated" but rather "open() returned a valid file descriptor (and not an error)". even if this is how POSIX described it, the behavior of updating *ctime* and not *mtime* doesnt make any real logical sense. as for the race condition, if you want to interpret "mark for update" to mean the times get updated on the fd but the file itself is delayed for updates, then simply changing to fstat() rather than stat() would address that concern. The assumptions in the Linux kernel build code are wrong; it should be using "touch", not "open with O_TRUNC". the assumption in the kernel build code is that the system is POSIX compliant See also the Single UNIX Specification: that is the spec ive been quoting all along -mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 7/9/07, Terry Lambert <tlambert(a)apple.com> wrote: <http://www.opengroup.org/onlinepubs/009695399/functions/open.html> This email sent to site_archiver(a)lists.apple.com

1 0

Re: EINVAL error when call pthread_attr_setstacksize
by Terry Lambert 10 Jul '07

10 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hello! I am trying to set thread stack size attribute. My code is stacksize = (PTHREAD_STACK_MIN); res = pthread_attr_setstacksize(&tattr, stacksize); if(res != 0) { printf("Failed to set thread stack size due to: %s\n", strerror(res)); return false; } However, I always got EINVAL error. I read the man page and some posts on the mailing list and knew that the stack size has to be a multiple of 4KB (4096). I tried but I just always got EINVAL error. Here is my system information 8.10.1 Darwin Kernel Version 8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386 i386 i386 -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Jul 9, 2007, at 3:32 PM, Heidi Zhou wrote: Any idea why? At a guess, you are declaring a static instance of a thread attribute, and assuming it is initialized by default, which it isn't. You must call pthread_attr_init() before calling pthread_attr_setstacksize() or any other attribute changes. This email sent to site_archiver(a)lists.apple.com

1 0

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Terry Lambert 09 Jul '07

09 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com thanks, ive posted it there as #5322203 -mike See also the Single UNIX Specification: <http://www.opengroup.org/onlinepubs/009695399/functions/open.html> -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Jul 9, 2007, at 12:09 PM, Mike Frysinger wrote: On 7/9/07, Shawn Erickson <shawnce(a)gmail.com> wrote: On 7/9/07, Mike Frysinger <vapier.adi(a)gmail.com> wrote: > On 6/12/07, Mike Frysinger <vapier.adi(a)gmail.com> wrote: > > i was tracking down a bug in the Linux kernel build system which seems > > to be caused by a bug in Darwin. the kernel build system creates a > > bunch of 0 byte files and then uses open(O_TRUNC) on them in order to > > update the st_mtime field (which triggers make to recreate some object > > files). on my mac mini, this all falls apart. > > is there somewhere i can file a bug report about this ? i'm really > not familiar with Darwin development methodologies <http://developer.apple.com/bugreporter/> There's already a bug filed on this, and it's currently "Not to be fixed". It's perfectly legitimate under POSIX to check the length of the file before attempting the truncation, and if the truncation would have no effect, not call into the VOP to do the truncation, thus not modifying the file. Specifically, the modification time will only be changed if the file is modified, which it is not, in this case (the POSIX mandate in the case of an actual truncation -- "successful completion" -- is "shall mark for update", rather than "shall update", in any event, so any code with this assumption built into it is inherently racy). The assumptions in the Linux kernel build code are wrong; it should be using "touch", not "open with O_TRUNC". This email sent to site_archiver(a)lists.apple.com

1 0

EINVAL error when call pthread_attr_setstacksize
by Heidi Zhou 09 Jul '07

09 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=XjytcqF1Z8HCrX6pwdrfd7wSx9bGDAZcoNFAL8GxMxSG2YeEOeHj2boq8Qd0YUg9DfSetQFa4TNC2sftfvCeCZpadaJAuiMr6ITqF3/xmogCYyZmxcZRu1u2E2Eo/jhvXcK7n8G7MV1N1Bzyd9kuiLJkoybv/T2YxltotPEu6cs=; Hello! I am trying to set thread stack size attribute. My code is stacksize = (PTHREAD_STACK_MIN); res = pthread_attr_setstacksize(&tattr, stacksize); if(res != 0) { printf("Failed to set thread stack size due to: %s\n", strerror(res)); return false; } However, I always got EINVAL error. I read the man page and some posts on the mailing list and knew that the stack size has to be a multiple of 4KB (4096). I tried but I just always got EINVAL error. Here is my system information 8.10.1 Darwin Kernel Version 8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386 i386 i386 Any idea why? Thank you in advance. Heidi ____________________________________________________________________________________ Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. http://mobile.yahoo.com/go?refer=1GNXIC _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

RE: Illegal Instruction
by Heidi Zhou 09 Jul '07

09 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=ltQxZbboTwehW2UH6zBPrsX8jipcvEQkONRqTx6nxPiqcl9Tazj2QUTqpApX5aE1OeT8DQFxAEcwqrj/fTMdH/kcNE1bnwhOXk4KY82ZNA/762khgwBuzcep98gok0R5jWDHXdHd5whhhZkui51YirU7jqVZH+0YckR9iamRC4Q=; Peter, Thank you. This works. Heidi --- Peter O'Gorman <peter(a)pogma.com> wrote: > On Mon, 2007-07-09 at 12:37 -0700, Heidi Zhou wrote: > > Norm, > > > > Thank you for the help. > > > > Your suggestion works! Now, I extend my test > program > > to generate a dynamic library. However, when I use > g++ > > to link, I got the following error > > > > $ g++ -g -Wall -dylib bin_osx/osServDarwin.o > > Not -dylib, -dynamiclib. You probably want to set > the install_name with > -install_name too. > > Peter > ____________________________________________________________________________________ Sick sense of humor? Visit Yahoo! TV's Comedy with an Edge to see what's on, when. http://tv.yahoo.com/collections/222 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

RE: Illegal Instruction (Norm Green)
by Norm Green 09 Jul '07

09 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

RE: Illegal Instruction (Norm Green)
by Peter O'Gorman 09 Jul '07

09 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Mon, 2007-07-09 at 12:37 -0700, Heidi Zhou wrote: > Norm, > > Thank you for the help. > > Your suggestion works! Now, I extend my test program > to generate a dynamic library. However, when I use g++ > to link, I got the following error > > $ g++ -g -Wall -dylib bin_osx/osServDarwin.o Not -dylib, -dynamiclib. You probably want to set the install_name with -install_name too. Peter _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0