October 2010 - Darwin-dev - lists.apple.com

Re: fsync while appending to a file
by Terry Lambert 25 Oct '10

25 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 25, 2010, at 2:27 PM, Joel Reymont wrote: > So how does this work when it works? > > How can you be writing to the buffers that you are trying to sync to disk? > > Thanks, Joel If you were trying to modify data in the underlying page, it could be "busy" and delay your writing, or it could be being written and delay your fsync(). Data is either committed or it isn't, depending on who gets in first, and you take your chances. Unless you do advisory file locking, or the descriptor in the other process was passed via UNIX domain socket or inherited as a result of a fork()/posix_spawn() (in which case they will share fileglobs, offset pointers, and so on, but maybe still not lock against each other). You could also be avoiding the buffer cache entirely, using direct I/O (F_NOCACHE; there are odd rules about already cached data not actually avoiding the cache in this case). Otherwise, your fd's are distinct and the fileproc that the fd points to has a separate fileglob, but a shared vnode. The fg_data in the fileglob, when it refers to a vnode, will point to the same vnode for every open instance of the same file (the file system will always vend the same vnode, if there is an existing reference already). For HFS, the vnode's v_data will point to the same cnode, and as I previously noted, it will ensure that the hfs_lock() is held over the call. Again, this is a property of HFS, and other file systems might not make as strong guarantees (HFS might not always act this way, either), so portable code should not depend on this behaviour. In any case, since you are racing one process into the write() for the append, and racing the other process into the fsync(), which one wins the race is going to determine if the new (dirty) pages have been attached to the file or not yet. Any dirty pages that are attached to the file before the fsync() will be committed to disk (cache), and the fsync() will not return until they are done being committed. If you need them committed to stable storage, you'd need to do an FUA to the drive, which you can do uding an fcntl(fd, F_FULLFSYNC), which is considerably expensive. Either way, unless you do locking, you are still racing to see who gets there first, and as I said previously, you shouldn't assume that the fsync() call is a serialization barrier for the write() call and vice versa, since they can race. That's FS implementation defined. Also for very large writes, the vn_rdwr() can loop breaking the write() operation up into smaller chunks in any case, so there's no strict guarantee of atomicity, only idempotence for the overall write(). In simple terms, if it broke the write up into three I/O's, then you could fsync() when anywhere from 0..3 of them had been written, depending on how far along in the loop it was when the other process raced in and fsync()'ed. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: fsync while appending to a file
by Joel Reymont 25 Oct '10

25 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=ZOZLJJugBnUUlWngD1H1hwpqRPIw+T814P9qD/CE0Go=; b=cD+dLo8AUdcotGx9P4+JpGHOruHpC31sOVlLGf9b/je6Ri9ifur/BhKEHAXPUTKXLz +UVdWN9AW/1anbJngnry4tX9hxrgXGwrt2m69jde4KBqOabZBJ7Wtrk2dtHSTVaNgtAm GZjkVoYqjDiNf1p4brrx81zvEoNLs3B+KISdU= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=sgPdiLhUTyfal1AFMnHS0+oNN1sfNUiZNS5m/73/MFInICgy+taK4jkygen/R7tk36 GD4O96YFjH5ejUNgpChzOEjSrTf2h1msXZJOK1sd/Nqd11VykhYx0gQvx7KlNbvZR32v HpFQlXzI7NdpYVT/pz7vRc+wZF8EeAOJXaci4= So how does this work when it works? How can you be writing to the buffers that you are trying to sync to disk? Thanks, Joel --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: fsync while appending to a file
by Terry Lambert 25 Oct '10

25 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 25, 2010, at 10:53 AM, Joel Reymont wrote: > Will fsync stall appending to a file, assuming both operations run concurrently and use different fds that point to the same file? > > Thanks, Joel It depends on the file system. For non-multithreaded filesystems, which are deprecated and unavailable on 64 bit kernels, yes. Otherwise, it's implementation defined on a per-file system basis. For HFS, also yes, since hfs_vnop_fsync() takes an hfs_lock() exclusive on the cnode while performing the operation. For UFS, the answer would have been no. I don't have an exact survey of answers for all the file systems supported by Mac OS X. That doesn't mean that if you are naturally racing that the append won't get in first, so if you need to ensure order of operation, for example, to ensure a database record gets written before a database index entry gets written as part of the standard two stage commit process, you should do locking anyway, rather than relying on implementation details of a particular FS. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

fsync while appending to a file
by Joel Reymont 25 Oct '10

25 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:content-type :content-transfer-encoding:subject:date:message-id:to:mime-version :x-mailer; bh=+QdBFekrTeS2PWx+ZLNcimIqnvGNrCJeCT3/ghoy0R8=; b=CY7hxuvT4lK887wblg/Ump4GEaGmy0LsxErhwC3/d2JQCq+4cB2+QVYfefG6QNzttW KEhR5zj6h4og3/H14xEuzy/9/vFcb05WyxAsUa3ay1RoDBcEHtFA6fzke0iLGzcj07MI 54KfLs5aSjlxOYssW99vF8LlCebBidA/ahm0E= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer; b=XlHGnzUDICdGaiT9xtcIlGVQMW77gJcBBPVBiizVQtY8bMDP2n/PrDwPN5tUQsa2OW 6d1e8ivS1qCZ4jzax17ceqsVyMV7mojgNRakaC8XZKhRJG179wfHBZCKQ5ZQ6zqxXYQs AxrkmAxuC2wnZepCwIV9JKYu8IULNbLUeLG54= Will fsync stall appending to a file, assuming both operations run concurrently and use different fds that point to the same file? Thanks, Joel --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Adding large file support
by Terry Lambert 25 Oct '10

25 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com off_t is 64 bit by default in Mac OS X, so it's not necessary to use special calls to obtain large file support. You will still need to use fseeko() instead of fseek(), unless you compile your code 64 bit (fseek() takes a long instead of an off_t because the standard requires it). I'm actually surprised off_t isn't 64 bit for Linux as well; it is for all of the BSDs. -- Terry On Oct 23, 2010, at 3:28 AM, Alex Reynolds wrote: > I have a C application I am trying to compile under Mac OS X 10.6.4, which includes large file support. The application builds correctly under Linux. > > Here is the Darwin kernel I am using: > > ------------------------- > $ uname -v > Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 > ------------------------- > > My gcc installation is as follows: > > ------------------------- > $ gcc --version > i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664) > ------------------------- > > My Makefile is as follows: > > ------------------------- > CC=gcc > CFLAGS=-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O3 -Wformat -Wall -pedantic -std=gnu99 > > all: myApp > rm -rf *~ > > myApp: myApp.o > ${CC} ${CFLAGS} myApp.o -lbz2 -o myApp > rm -rf *~ > > clean: > rm -rf *.o myApp > ------------------------- > > The issue is that my application makes calls to "fseeko64" and "fopen64", and it uses the "off64_t" type for offsets. When I attempt to build, I get the following related warnings and errors: > > ------------------------- > $ make myApp > gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O3 -Wformat -Wall -pedantic -std=gnu99 -c -o myApp.o myApp.c > myApp.c: In function ‘extractData’: > myApp.c:119: warning: implicit declaration of function ‘fseeko64’ > myApp.c:119: error: ‘off64_t’ undeclared (first use in this function) > myApp.c:119: error: (Each undeclared identifier is reported only once > myApp.c:119: error: for each function it appears in.) > myApp.c: In function ‘extractMetadata’: > myApp.c:305: warning: implicit declaration of function ‘fopen64’ > myApp.c:305: warning: assignment makes pointer from integer without a cast > ------------------------- > > My code builds without errors under Linux. What changes can I make to the source code to add large file support when building under Darwin? > > Thanks for your advice. > > Regards, > Alex _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/darwin-dev/tlambert%40apple.com > > This email sent to tlambert(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

TRIM for SSD in our near future
by Joel Reymont 24 Oct '10

24 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:content-type :content-transfer-encoding:subject:date:message-id:to:mime-version :x-mailer; bh=mgU/dp6EES7eUK6KCEFwXdgZ2CwdLAISkXQ76YT9q44=; b=p+mQIP6wbmmgJUih8+uzcFEoH7K8HECGM7bMlAXVK+SAIMpTSFel1/L7TMXO25EqS3 +zSbKWP5NHA6d6s8fpj0JtbfK5mgiwftTU/d2fGIAlSqLJzBPqdXSg8SwRaSUkFRBEbq sf2/fGdkAMfcUrIE3toix3FLpAIvPi9I3TihM= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer; b=Hz+v1QWKv2vXB8krQCEcMkOEQuU49INDpn7ZLV3GJbUYccMlc3UE0kFNQXOkHETL2o HqjJliHfuXKDmIeeRst+0MFgKbx3aEwvZzy+98TVniGn0r5DCxVtqvGEKIv5jgWvUgh7 acQ8E2oBAueV14CU2UYraSu1+gv+OneJrPYEc= Can anyone hint, privately perhaps, if TRIM is coming to Mac OSX in the near future? Thanks, Joel --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Adding large file support
by Rustam Muginov 23 Oct '10

23 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=pdtL2Z0ELO8TEbHphihqbiKNMrTqTzWwOPLWV070ex8=; b=OhTzbFEpHE5tUz+B2lXkEl7R6em8lRp9cWOXg3xsGnhpIcdmjr+x0MSDaJtSeir+xu x8ePXU/uk7K3aPa/ZW7RQfWvXgz/0VLNSdR6wsVAtZjbD68f8LHQFM0yGFjdKYJ+iCIB 0bpLEplM30I3hGDXebCctEuNqlodR84SWoSp4= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=VBxU+ondoTkHkuS/v+U/1HPxVr5TkRfRmJgyBGGSRdU/GYDabxOmb75aV5TR7y8rfz MMCbkZHCjkS4vDSABk/RmojKgm3TE9Bl7cLP7V7wQtLy3kfQL74yr+YSW9D6JyXvxX/X 1xB4kVrpLNmefMf8e0M+dBnDvCTG4SdwDnz+Y= AFAIC where are no "fseeko64/fopen64" in C standard. For 64-bit offset, use int fseeko(FILE *stream, off_t offset, int whence); for definition of off_t, use #include <sys/types.h> You can read more in Mac OS X man pages. On Oct 23, 2010, at 2:28 PM, Alex Reynolds wrote: > I have a C application I am trying to compile under Mac OS X 10.6.4, which includes large file support. The application builds correctly under Linux. > > Here is the Darwin kernel I am using: > > ------------------------- > $ uname -v > Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 > ------------------------- > > My gcc installation is as follows: > > ------------------------- > $ gcc --version > i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664) > ------------------------- > > My Makefile is as follows: > > ------------------------- > CC=gcc > CFLAGS=-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O3 -Wformat -Wall -pedantic -std=gnu99 > > all: myApp > rm -rf *~ > > myApp: myApp.o > ${CC} ${CFLAGS} myApp.o -lbz2 -o myApp > rm -rf *~ > > clean: > rm -rf *.o myApp > ------------------------- > > The issue is that my application makes calls to "fseeko64" and "fopen64", and it uses the "off64_t" type for offsets. When I attempt to build, I get the following related warnings and errors: > > ------------------------- > $ make myApp > gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O3 -Wformat -Wall -pedantic -std=gnu99 -c -o myApp.o myApp.c > myApp.c: In function ‘extractData’: > myApp.c:119: warning: implicit declaration of function ‘fseeko64’ > myApp.c:119: error: ‘off64_t’ undeclared (first use in this function) > myApp.c:119: error: (Each undeclared identifier is reported only once > myApp.c:119: error: for each function it appears in.) > myApp.c: In function ‘extractMetadata’: > myApp.c:305: warning: implicit declaration of function ‘fopen64’ > myApp.c:305: warning: assignment makes pointer from integer without a cast > ------------------------- > > My code builds without errors under Linux. What changes can I make to the source code to add large file support when building under Darwin? > > Thanks for your advice. > > Regards, > Alex _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/darwin-dev/rmuginov%40gmail.com > > This email sent to rmuginov(a)gmail.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Adding large file support
by Alex Reynolds 23 Oct '10

23 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com I have a C application I am trying to compile under Mac OS X 10.6.4, which includes large file support. The application builds correctly under Linux. Here is the Darwin kernel I am using: ------------------------- $ uname -v Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 ------------------------- My gcc installation is as follows: ------------------------- $ gcc --version i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664) ------------------------- My Makefile is as follows: ------------------------- CC=gcc CFLAGS=-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O3 -Wformat -Wall -pedantic -std=gnu99 all: myApp rm -rf *~ myApp: myApp.o ${CC} ${CFLAGS} myApp.o -lbz2 -o myApp rm -rf *~ clean: rm -rf *.o myApp ------------------------- The issue is that my application makes calls to "fseeko64" and "fopen64", and it uses the "off64_t" type for offsets. When I attempt to build, I get the following related warnings and errors: ------------------------- $ make myApp gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O3 -Wformat -Wall -pedantic -std=gnu99 -c -o myApp.o myApp.c myApp.c: In function ‘extractData’: myApp.c:119: warning: implicit declaration of function ‘fseeko64’ myApp.c:119: error: ‘off64_t’ undeclared (first use in this function) myApp.c:119: error: (Each undeclared identifier is reported only once myApp.c:119: error: for each function it appears in.) myApp.c: In function ‘extractMetadata’: myApp.c:305: warning: implicit declaration of function ‘fopen64’ myApp.c:305: warning: assignment makes pointer from integer without a cast ------------------------- My code builds without errors under Linux. What changes can I make to the source code to add large file support when building under Darwin? Thanks for your advice. Regards, Alex _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: ASL: log as a circular buffer
by Marc Majka 14 Oct '10

14 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: asl: unsetting keys not working
by Marc Majka 14 Oct '10

14 Oct '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0