Mailing Lists: Apple Mailing Lists
Image of Mac OS face in stamp
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Strange behaviour of mmap() in OS X

On Saturday, June 21, 2003, at 6:06 AM, Marcel Weiher wrote:
Note that other UNIXes I tested (Linux, IRIX) seem to handle mmap() more gracefully and the system remains fully usable during and after the mmap(). The amount of swapping is minimal.

There shouldn't really be any swap-out at all.

Any pageout algorithm that automatically punishes mapped file pages (simply because they don't have to be cleaned before recycling in many cases) is inherently unfare. Especially since mapped file pages are already "punished" on the front-end of the page replacement algorithm. Consider the fact that you can zero fill lots of pages in the amount of time it takes to read just one back from a disk (even after read-ahead type algorithms are applied). So we already favor adding anonymous pages _into_ the page pool. If we also favored keeping anonymous pages in the pool longer (because we almost always have to swap them to replace them), then there would be a very heavy bias towards those types of pages.

So, without explicit intervention/hinting, I wholeheartedly disagree with your assertion about "there shouldn't really be any swapping." If the next likely page to replace needs to be swapped in order to bring in the next page needed from a mapped file, we should do the swapping. Of course, if there is hinting...we should be able to do a better job of choosing a page to replace (see below).


This was not due to the amount of free memory: The Linux box has about 256 MB RAM with ca. 150 MB free, the IRIX machine has only 64 MB with ca. 30 MB free. The Mac (a PowerBook 12") has 640 MB RAM, 180 MB were unused (inactive). All data was gathered with 'top'.

I know that I can give the VM manager hints about the memory usage (I use it sequentially), but memadvise() seems to be broken in OS X. I have this impression from previous posts to several mailing lists and my own experience.

The BSD madvise() API is mostly forward-looking (and all future effects on the memory as a result of the call are optional/advisory only). So, even if it does nothing, it's not, technically, broken. And we do more than nothing when these are called. It's just that we not be aggressive as some other vendors. But we had some reason to our madness:

Specifically, in your case, marking something as "sequential" still doesn't give us quite enough information to avoid doing "really stupid" things in some cases. That is, how far back from the current "fault" should we start deactivating pages? If it's too far back, we still spill over the available memory and at least have to swap some. If it's not far enough back, the application could re-access the page right after we deactivated it. That's because while pre-fetching in the sequential case, we rarely are working in the VM system with the same page the thread is currently working on. We could add some artificial overhead to cause faults and detect where the thread "is". But that's assuming a straight [single] linear progression through the pages. With vector code, etc, you often make several passes through small[ish] data chunks and then move on. So, "sequential" isn't all that meaningful these days.

Instead, we decided to rely on specific "past-looking" feedback. The msync(MS_INVALIDATE) call can be used to inform us which pages you absolutely don't need anymore. It's very similar to madvise(MADV_DONTNEED). But it has much more predictable scheduling behavior in the face of actually finding dirty pages in the range. That is: whose thread is used to do the cleaning to make the pages available? Many assume the madvise() call will return somewhat immediately, and the cleaning of any pages will happen asynchronously. But that asynchronous behavior can affect the rest of your media scheduling in unpredictable ways. If you use the synchronous msync() call, you have greater control over all that.

So, we concentrated on the synchronous (and therefore much more predictable) msync() first.

--Jim
_______________________________________________
darwin-development mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-development
Do not post admin requests to the list. They will be ignored.
References: 
 >Re: Strange behaviour of mmap() in OS X (From: Marcel Weiher <email@hidden>)



Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Contact Apple | Terms of Use | Privacy Policy

Copyright © 2007 Apple Inc. All rights reserved.