February 2005 - Darwin-dev - lists.apple.com

Re: Various tools update request
by Alex v.Below 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Am 21.02.2005 um 19:48 schrieb Jordan Hubbard: Bye Alex _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… People have cited cvs as an example of "neglect" in this discussion when, in fact, we've wanted to update it for years but for a semi-controversial "fork" over the "tar wrappers" feature, which it took us years to resolve in a manner that would let us move forward again (and we have). Yes, I have talked to Apple Engineers at WWDC about this, and I know this semi-controversial dispute. Good to see it resolved! Also I would like to add that my overall experience with bugreport.apple.com has been exellent. Filing a meaningful bugreport will usually fix the problem in time. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Various tools update request
by Jordan Hubbard 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Various tools update request
by Jordan Breeding 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Various tools update request
by Jordan Hubbard 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: bad fsync? (A.M.)
by James Berry 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dominic, From: asko.kauppi(a)sci.fi Subject: [sqlite] sluggish operation on os x? Date: February 21, 2005 12:44:03 AM PST To: sqlite-users(a)sqlite.org Reply-To: sqlite-users(a)sqlite.org .... From: cking(a)mumbo.ca Subject: Re: [sqlite] sluggish operation on os x? Date: February 21, 2005 9:40:27 AM PST To: sqlite-users(a)sqlite.org Reply-To: sqlite-users(a)sqlite.org ck On Feb 19, 2005, at 5:59 PM, Dominic Giampaolo wrote: MySQL makes the following claim at: http://dev.mysql.com/doc/mysql/en/news-4-1-9.html "InnoDB: Use the fcntl() file flush method on Mac OS X versions 10.3 and up. Apple had disabled fsync() in Mac OS X for internal disk drives, which caused corruption at power outages." First of all, is this accurate? A pointer to some docs or a tech note on this would be helpful. The comments about fsync() are wrong... On MacOS X, fsync() always has and always will flush all file data from host memory to the drive on which the file resides. The behavior of fsync() on MacOS X is the same as it is on every other version of Unix since the dawn of time (well, since the introduction of fsync anyway :-). I believe that what the above comment refers to is the fact that fsync() is not sufficient to guarantee that your data is on stable storage and on MacOS X we provide a fcntl(), called F_FULLFSYNC, to ask the drive to flush all buffered data to stable storage. Let me explain in more detail. With fsync() even though the OS writes the data through to the disk and the disk says "yes I wrote the data", the data is not actually on permanent storage. Unless you explicitly disable it, all disks have a write buffer which holds data you've written. The disk buffers the data you wrote until it decides to flush it to the platters (and the writes may not be in the order you wrote them). If you lose power or the system crashes before the data is written, you can wind up in a situation where only some of your data is actually on disk. What is worse is that even if you write blocks A, B and C, call fsync() and then write block D you may find after rebooting that blocks A and D are on disk but B and C are not (in fact any ordering of A, B, C, and D is possible). While this may seem like a rare case it is not. In fact if you sit down and pull the plug on a system you can make it happen in one or two plug pulls. I have even gone so far as to watch this behavior with a logic analyzer on the ATA bus: I saw the data for two writes come across the ATA cable, the drive replied and said the writes were successful and then when we rebooted the data from the second write was correct on disk but the data from the first write was not. To deal with this we introduced the F_FULLFSYNC fcntl which will ask the drive to flush all of its buffered data to disk. When an app needs to guarantee that data is on disk it should use F_FULLFSYNC. In most cases you do not need such a heavy handed operation and fsync() is good enough. But in an app like a database, it is essential if you want transactional integrity. Now, a little bit more detail: on ATA drives we implement F_FULLFSYNC with the FLUSH_TRACK_CACHE command. All drives sold by Apple will honor this command. Unfortunately quite a few firewire drive vendors disable this command and do not pass it to the drive. This means that most external firewire drives are not reliable if you lose power or the system crashes. We can't work-around that unless we ask the drive to disable the write cache completely (which hurts performance quite badly -- and even that may not be enough as some drives will ignore that request too). So in summary, I believe that the comments in the MySQL news posting are slightly confused. On MacOS X fsync() behaves the same as it does on all Unices. That's not good enough if you really care about data integrity and so we also provide the F_FULLFSYNC fcntl. As far as I know, MacOS X is the only OS to provide this feature for apps that need to truly guarantee their data is on disk. Hope this clears things up. --dominic _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/james%40jberry.us This email sent to james(a)jberry.us _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Thanks for a great explanation. A similar thread has come up on the sqlite list, claiming that using F_FULLFSYNC leads to dramatic performance decreases on Darwin over other platforms. Any thoughts on these issues? Finally getting my SQLite3 code working, I'm experiencing awfully slow performance when writing individual data on OS X. Linux (Ubuntu) is lightning fast, Win32 is.. tolerable, but OS X really crawls. I've tried both with built-in sources, and the new SQLite3 fink module. Same behaviour. I noticed this as well, so I profiled my call and found sync was taking forever. I removed the following fcntl call, rc = fcntl(fd, F_FULLFSYNC, 0);. Performance was back to normal. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Threads problems
by Jonas Maebe 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 21 feb 2005, at 18:28, Matthew Jenkins wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Is there something fundamental that I'm missing? Yes, either you should create your threads in a detached state (or detach them inside their runloop), or you should pthread_join() them in the main program. This email sent to site_archiver(a)lists.apple.com

1 0

Threads problems
by Matthew Jenkins 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi, I am having "issues" with threads on darwin and memory consumption. Is there something fundamental that I'm missing? Cheers, Matt -- Darwin dannii.hq.inty.net 7.8.0 Darwin Kernel Version 7.8.0 #include <stdio.h> #include <pthread.h> void mythread() { printf("moo\n"); } int main() { int i; pthread_t q; for(i=0; i<100; i++) { pthread_create(&q,NULL,(void *)&mythread,NULL); sleep(5); } return(0); } Scanned for viruses by MailDefender _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… It seems that with every thread I create about 4k of resident memory is allocated. That memory never seems to be reclaimed at thread termination. My little test program, which creates 100 threads at 5 second intervals is posted below. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Various tools update request
by Eric Seidel 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com http://www.opensource.apple.com/darwinsource/WWDC2004/ Thanks! -eric (former darwin hacker) On Feb 21, 2005, at 3:37 AM, Adriaan van Os wrote: - bison An old version gets installed. Bison 2.0 has been released <ftp://ftp.gnu.org/gnu/bison/bison-2.0.tar.gz> - help2man Unfortunately, this tool is not installed with Xcode or OS X/Darwin (it can be configured with --enable-nls=no to prevent build problems) <ftp://ftp.gnu.org/gnu/help2man/help2man-1.35.1.tar.gz> Regards, Adriaan van Os _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/eseidel%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… You might also wish to verify your complaints against the latest publicly released tiger sources: Yes, they're 9 months old by now... but you may find newer versions there than in the 10.3.x sources. Please make sure that you file *individual* bugs for each project upgrade request. Including the link to the website and tarball (along with any patches necessary to make the latest and greatest compile). I request that Apple updates some of the tools distributed with OS X/Darwin and Xcode - sed FreeBSD sed gets installed, but it is crippled (doesn't recognize regular expressions) GNU sed is preferred <ftp://ftp.gnu.org/gnu/sed/> - flex An old version of flex gets installed. Please note that flex is now on sourceforge (previously on gnu ftp servers) <http://sourceforge.net/project/showfiles.php?group_id=72099> - configure problems There is a bug in Mac OS X that causes problems when configuring many software packages (see e.g.<http://lists.gnu.org/archive/html/bug-coreutils/2004-08/ msg00039.html>). As a result of this bug, the coreutils command "uname -p" reports "unknown". This confuses config.guess. This email sent to eseidel(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Various tools update request
by Jonas Maebe 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 21 feb 2005, at 13:44, Axel Andersson wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Last time I checked, Darwin userland was still BSD. No, it's a mix of BSD and GNU (e.g. the default make is GNU make since 10.1 or 10.2). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Various tools update request
by Axel Andersson 21 Feb '05

21 Feb '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Feb 21, 2005, at 12:37, Adriaan van Os wrote: On Feb 21, 2005, at 12:46, Gregory Seidman wrote: - make GNU make 3.80 has been out for over two years. Note that it may be desirable to take patches from Debian, which fix some issues. The Debian-patched make 3.80 compiles cleanly under MacOS 10.3.x. -- Axel Andersson axel(a)zankasoftware.com http://www.zankasoftware.com/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… - sed FreeBSD sed gets installed, but it is crippled (doesn't recognize regular expressions) GNU sed is preferred <ftp://ftp.gnu.org/gnu/sed/> Last time I checked, Darwin userland was still BSD. Please don't try to cripple it with GNU-specific utilities. And regardless, this is not a very suitable channel for requesting updates. http://bugreporter.apple.com/ is. This email sent to site_archiver(a)lists.apple.com

1 0