Jim Magee writes:
On Wednesday, September 11, 2002, at 11:49 AM, Andrew Gallatin wrote:
I could make all uses APPLE_API_PRIVATE headers go away in my driver
(OS-Bypass cluster interconnect --
http://www.myri.com/scs/macosx.html) if somebody would just fix the
VM system so that it wouldn't block a user process indefinatly if said
process attempts to free memory locked via IOMemoryDescriptor's
prepare function. But I couldn't seem to get anybody interested in
changing this behaviour in the past...
I think I even exchanged some mail with you on this subject last time.
Yes, I think you did. To refresh, the problem happens when an app has registered memory for os-bypass IO via an ioctl into our driver. The memory is locked via prepare. In the normal course of operations, a task will deregister the memory (and our driver will unprepare it, thus unlocking it) prior to freeing it. The problem occurs when a buggy applications neglects to deregister the memory prior to freeing it. This is a bug in the application (they've violated our API). But the resulting behaviour (process is unkillable, sleeping waiting for memory to be unprepared; our driver won't unprepare the memory because it has no idea that it should) is sub optimal.
Doing what you ask would require "prepare" to create and hold its own
reference to the underlying memory objects rather than "glom-ing" on to
the user's reference (leaving the user free to release his own with
impunity). This could be done, but it would slow prepare down even
more in its current form (and it is in dire need of a performance boost
already). In fact, we are in the process of investigating possible
enhancements to the implementation. The leading design does involve
burrowing all the way down to the object, so we do have the opportunity
to think about taking our own references again. This would allow
unprepare to be much more efficient, and subsequent prepares in the
same object could also be quicker. But is also allows a possible
redress of your issue.
Excellent! Also, memory locking on MacOSX is currently slower than on any other unix-like platform we support. It would be good to see it sped up.
The real question is: Does anything depend upon that portion of the
address space being locked down while I/O is in progress? It would be
difficult to see how they could, because the prepare gets released
before their notion of an I/O operation is marked as complete. But
it's just something to investigate.
Nothing depends on that part of the address space being locked down. However, the physical memory (formerly) backing the portion of the apps virtual address space which it has just freed cannot be recycled. Otherwise we risk either corrupting memory if those pages end up belonging to some other task. Ideally, the memory would be removed from the app's address space when it called free(), and the physical pages would held in limbo, only freed back to the system when we unprepared it and some reference count dropped to zero. This is what our linux driver does (by hand, in our driver, since there is no api for it. 300+ lines of code which change frequently. Ick). Actually, I suppose it could be freed if you had a sophisticated IOMMU and could remap those addresses to the bit bucket when they were accessed via dma. ;)
--Jim
PS: I think, last we left it, we also talked about changing the timing
between file-descriptor closing and VM cleaning. We intentionally
clean up the Mach port space (which IOKit UserClient interfaces
typically use) before the VM space so that UserClient memory
descriptors would be freed before attempting to free the memory in an
exiting process. Closing file descriptors should happen even before
that (to allow file-descriptor-based drivers to release their holds on
memory). While this may not be necessary, if we do the work above, it
is still the cleaner way to go.
This like what I was asking for... Has the behaviour changed? I haven't looked into this since I wrote my reaper thread work around, which uses APPLE_API_PRIVATE apis. The problem I was running into earlier was due to the process's VM space being cleaned up _prior_ to its file-descriptors being closed. The effect would be similar to what I talk about above -- the VM cleanup code frees registered memory, leading to a deadlock. Closing the file-descriptors first avoids the deadlock, because our driver deregisters the app's memory at this point. Thanks! Drew _______________________________________________ darwin-kernel mailing list | darwin-kernel@lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.