site_archiver@lists.apple.com Delivered-To: Darwin-kernel@lists.apple.com We have a problem with a ring buffer program which memory maps a large file. [ ... ] We can evade the problem by initializing the buffer file before starting the ring buffer program, but this is operationally difficult -- and it hasn't been necessary on other platforms. The problem appears to be related to how quickly the ring buffer program is writing to the buffer file -- if it's slow enough and the buffer file is small enough, we typically don't see the problem. It seems to need multiple threads writing in the operational program. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.a... On Jul 7, 2010, at 12:37 PM, Daniel Quinlan wrote: I'm not certain this is the right list for this problem, but it seems to me to be a kernel problem of some sort. It's rather long, but I've tried to be comprehensive. I've also submitted this as a bug to the apple bug reporter. HFS does not support sparse files, so you are waiting for it to allocate backing pages. On the other operating systems, the FS's you are using apparent support sparse files, and therefore the backing pages are being lazily allocated for the sparse file control structures, which are allocable quickly. Technically, you still have a race condition on those other machines, you are just lucky that the order you start writing is the same as the order in which the control structures are allocated. That is you win the race because your use is such that the control structures themselves are in place by the time you get to any region of the file governed by the control structures in question. It would really be best on all platforms for you to preallocate your file, or at least the control structures. On Mac OS X you can use UFS (if it's a very old Mac OS X) or UDF (if it's a modern one) since both those filesystem types support sparse files. This is still racy (as noted above), but you would be in the same boat as you are on the other filesystems. PS: You do not get an aggregate performance win for using a sparse file. You win on startup time, but what you win there, you lose in runtime, as the operations are stalled waiting for backing store allocation whenever you hit a file region that does not already have backing store pre-allocated. If runtime performance is more important than startup performance, you may want to take that into account and write a byte in the filesystem blocks on block boundaries after allocating them on the other operating systems. PPS: I would minimally put an interlock there, such as opening the file initially O_EXCL, to block the second process' open of the file to prevent the file being used by the second process before it's ready, no matter what else you do. This email sent to site_archiver@lists.apple.com