Re: Thoughts about streaming from disk and VM
Re: Thoughts about streaming from disk and VM
- Subject: Re: Thoughts about streaming from disk and VM
- From: Herbie Robinson <email@hidden>
- Date: Thu, 13 Mar 2003 04:32:01 -0500
At 1:02 PM -0600 3/12/03, Eric Huffman wrote:
Multiple Files (VM) -
Under a VM OS, the above approach for multiple files seems to have
non-optimal runtime behavior since the VM manager has to perform
additional memory management for the buffer memory (e.g. possibly paging
out memory for the allocation of the buffers, possibly paging in the
buffers if they were paged out since the last file IO, and possibly
paging out the buffers after the file IO).
I haven't actually tried this in OS X; so, I may be blowing smoke,
but many Unix implementations optimize the case where the memory and
disk addresses both start on a page boundary.
I believe that you will have to wire the buffers or you will not get
reliable behavior; so, paging shouldn't come into play. And yes,
this will take a lot of memory.
Another approach involves mapping the files into memory (e.g. using
mmap). This avoids most of the mentioned overhead related to the VM
manager and the file buffers. However, this is not optimal for
sequential disk access since the files will be read in page size chunks.
You have perceived the problem here. From the behavior of other
apps, it looks like you will need to queue up at least 1 second of
I/O at a time. That's around 150K bytes per I/O. The idea is to
queue up a lot of I/O and allow the disk driver to optimize the seek
ordering.
One other thing that has been posted is that the disk software
doesn't really support asynchronous transfers. The Carbon API does
define them, but apparently they are implemented by starting threads.
This means that one probably needs one thread per file (or maybe one
thread per channel if multiple channels are reading from the same
file.
An interesting addition is that of heuristically and occasionally
"pre-warming" the mapped memory by touching the mapped memory in chunks
greater than a page size. This helps in driving sequential access to
the disk at the expense incurred by the "pre-warming" but still does not
guarantee that the memory that will need to be read later will still be
in core memory.
The latter problem can only be cured by wiring the buffers. You
can't be paging if you are pushing the disk to the limit doing the
I/O!
This will only help if you can "pre-warm" a large chunk of memory and
the implementation of the pre-warming passes the full range of pages
all the way to the disk driver.
--
-*****************************************
**
http://www.curbside-recording.com/ **
******************************************
_______________________________________________
coreaudio-api mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/coreaudio-api
Do not post admin requests to the list. They will be ignored.