Re: Disk streaming library
Re: Disk streaming library
- Subject: Re: Disk streaming library
- From: Patrick Shirkey <email@hidden>
- Date: Fri, 29 Jun 2012 11:04:46 +0200 (CEST)
- Importance: Normal
On Fri, June 29, 2012 9:50 am, Ross Bencina wrote:
> On 29/06/2012 12:54 AM, Paul Davis wrote:
>> this question is, well, a bit *undefined*.
>
> Agreed.
>
>> but wait, can't you do
>> better that the OS by rolling your own? say, by opening the file in
>> "direct" mode such that the buffer cache is not used, and then doing
>> intelligent caching in an application/data-specific way?
>
> On Windows you can better in my experience.
>
> I'm not sure what metrics are being applied here but there are a few to
> consider: Even if the same disk throughput can be achieved using the
> native file system cache, a workload-specific prefetch system might use
> less CPU, give lower read latency, use less RAM etc.
>
> With an N+M heuristic I would suggest that M needs to be controlled
> based on the workload (ie file read-rate). Application level
> caching/prefetch may still be needed for non-linear reads (ie looping),
> or the "keep the first few seconds in RAM" thing.
>
> One thing you *do* want is to have as many io-ops in flight at once.
> Afaik the interfaces for doing this are completely different accross
> Linux, MacOS and Windows.
>
>
>> well, good luck with that. Oracle certainly pulls this off with their
>> databases, but there have been many, many studies where people have
>> tried to do better than the buffer cache and discovered that in real
>> world scenarios, they can't.
>
> Links please?
>
> There are other reasons to roll your own: asynchrony, determinism, i/o
> prioritisation, portability to platforms that don't meet your
> Linux-centric assumptions.
>
>
>> if you were to accept that to be the end of
>> the story (it probably isn't), then on OS X at least, you wouldn't plan
>> on using any "disk streaming engine" at all - you'd just do regular
>> system calls to read/write and let the OS take care of the rest.
>>
>> to get any kind of an answer to this question, i suspect you need to
>> describe in more detail what you mean by "a high performance disk
>> streaming solution".
>
> Agreed. And I also agree with the "simplest is best" approach until
> proven otherwise by practical experiment.
>
> The main thing is to decouple computation from i/o so you can keep the
> io-op pipeline full. Anything that forces you to synchronously
> interleave computation with io is not so good for this (*cough*
> libsndfile).
>
Is Erik round to defend that?
Just wondering if the assessment is any different on iOS?
--
Patrick Shirkey
Boost Hardware Ltd
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden