Re: Disk streaming library
Re: Disk streaming library
- Subject: Re: Disk streaming library
- From: Ross Bencina <email@hidden>
- Date: Fri, 29 Jun 2012 17:50:44 +1000
On 29/06/2012 12:54 AM, Paul Davis wrote:
this question is, well, a bit *undefined*.
Agreed.
but wait, can't you do
better that the OS by rolling your own? say, by opening the file in
"direct" mode such that the buffer cache is not used, and then doing
intelligent caching in an application/data-specific way?
On Windows you can better in my experience.
I'm not sure what metrics are being applied here but there are a few to
consider: Even if the same disk throughput can be achieved using the
native file system cache, a workload-specific prefetch system might use
less CPU, give lower read latency, use less RAM etc.
With an N+M heuristic I would suggest that M needs to be controlled
based on the workload (ie file read-rate). Application level
caching/prefetch may still be needed for non-linear reads (ie looping),
or the "keep the first few seconds in RAM" thing.
One thing you *do* want is to have as many io-ops in flight at once.
Afaik the interfaces for doing this are completely different accross
Linux, MacOS and Windows.
well, good luck with that. Oracle certainly pulls this off with their
databases, but there have been many, many studies where people have
tried to do better than the buffer cache and discovered that in real
world scenarios, they can't.
Links please?
There are other reasons to roll your own: asynchrony, determinism, i/o
prioritisation, portability to platforms that don't meet your
Linux-centric assumptions.
if you were to accept that to be the end of
the story (it probably isn't), then on OS X at least, you wouldn't plan
on using any "disk streaming engine" at all - you'd just do regular
system calls to read/write and let the OS take care of the rest.
to get any kind of an answer to this question, i suspect you need to
describe in more detail what you mean by "a high performance disk
streaming solution".
Agreed. And I also agree with the "simplest is best" approach until
proven otherwise by practical experiment.
The main thing is to decouple computation from i/o so you can keep the
io-op pipeline full. Anything that forces you to synchronously
interleave computation with io is not so good for this (*cough* libsndfile).
Ross.
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden