Re: Disk streaming library
Re: Disk streaming library
- Subject: Re: Disk streaming library
- From: Philippe Wicker <email@hidden>
- Date: Fri, 29 Jun 2012 11:04:11 +0200
On 29 Jun 2012, at 09:50, Ross Bencina wrote:
> On 29/06/2012 12:54 AM, Paul Davis wrote:
>> this question is, well, a bit *undefined*.
>
> Agreed.
>
>> but wait, can't you do
>> better that the OS by rolling your own? say, by opening the file in
>> "direct" mode such that the buffer cache is not used, and then doing
>> intelligent caching in an application/data-specific way?
>
> On Windows you can better in my experience.
>
> I'm not sure what metrics are being applied here but there are a few to consider: Even if the same disk throughput can be achieved using the native file system cache, a workload-specific prefetch system might use less CPU, give lower read latency, use less RAM etc.
>
> With an N+M heuristic I would suggest that M needs to be controlled based on the workload (ie file read-rate). Application level caching/prefetch may still be needed for non-linear reads (ie looping), or the "keep the first few seconds in RAM" thing.
Yes. We already reached the conclusion that we'll have to cache data around the loop locators. The "keep the first few seconds in RAM" thing is - to my knowledge - the only way to guarantee a low latency (e.g. when a note on is played).
>
> One thing you *do* want is to have as many io-ops in flight at once. Afaik the interfaces for doing this are completely different accross Linux, MacOS and Windows.
I'm not aware of all the APIs that could enable that, but I'd think that an asynchronous read API is needed to give the OS (the disk driver maybe) a chance to reorganise (reschedule) a bunch of IO requests.
>
>
>> well, good luck with that. Oracle certainly pulls this off with their
>> databases, but there have been many, many studies where people have
>> tried to do better than the buffer cache and discovered that in real
>> world scenarios, they can't.
>
> Links please?
>
> There are other reasons to roll your own: asynchrony, determinism, i/o prioritisation, portability to platforms that don't meet your Linux-centric assumptions.
>
>
>> if you were to accept that to be the end of
>> the story (it probably isn't), then on OS X at least, you wouldn't plan
>> on using any "disk streaming engine" at all - you'd just do regular
>> system calls to read/write and let the OS take care of the rest.
>>
>> to get any kind of an answer to this question, i suspect you need to
>> describe in more detail what you mean by "a high performance disk
>> streaming solution".
>
> Agreed. And I also agree with the "simplest is best" approach until proven otherwise by practical experiment.
>
> The main thing is to decouple computation from i/o so you can keep the io-op pipeline full. Anything that forces you to synchronously interleave computation with io is not so good for this (*cough* libsndfile).
Hmmm. The need of data on all the streamed samples is uncorrelated between the samples. I mean that you need to start streaming a sample (or a group of samples) when a note ON (or a similar event) is received. From the plug-in point of view these events are random, unpredictable. Also the rate at which a sample has to be read depends on a lot of parameters (the note value, the pitch modulation…). Disk read commands have their origin in the audio callback because this is where events are processed. It may happen that a few read commands are sent to the worker thread at roughly the same time. Do you mean then that we should wait for the completion of all the asynchronous IOs (assuming an asynchronous disk API) before processing the raw sample data (conversion and SRC)?
>
> Ross.
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden