Apple

> Many readers of the list will know who I am, but for those who don't let me start by just mentioning that I'm the lead developer and designer of Ardour, an open source GPL'ed DAW for Linux, OS X and Windows, and also the original author of JACK, a cross-platform API for low latency, realtime audio and MIDI, including inter-application and network routing.
>
> Over in Ardour-land, we've wrestled for a couple of years now with less than ideal support for OS X and are just finally beginning to resolve many/most of the areas where we were falling down.
>
> But there's an area where we've really run into a brick wall, despite the extremely deep and amazingly talented pool of people we have as developers, people who know *nix operating systems inside out and upside down.
>
> That area is: getting acceptable disk I/O bandwidth on OS X when reading many files. For comparison: we can easily handle 1024 tracks on a single spinning disk on Linux. We can get close to similar performance on Windows. We can even (gasp!) get in the same ballpark on Windows running as a guest OS on OS X (i.e. the entire Windows filesystem is just one file from OS X's perspective).
>
> But on OS X itself, we are unable to get consistent, reliable performance when reading many files at once. Our code is, of course, entirely cross-platform, and so we are not using any Apple specific APIs (or Linux-specific or Windows-specific). The same code works well for recording (writing) files (though OS X does still perform worse than other platforms).
>
> We've used various dtrace-based tools to try to analyse what is going on: these only confirmed for us that we have a problem, but provided no insights into what.
>
> Things became so inexplicable that we decided to break out the problem from Ardour itself and write a small test application that would let us easily collect data from many different systems. We've now tested on the order of a half-dozen different OS X systems, and they all show the same basic bad behaviour:
>
> * heavy dependence of sustained streaming bandwidth on the number of files being read
> (i.e. sustained streaming bandwidth is high when reading 10 files, but can be very low
> when reading 128 files; This dependence is low on Windows and non-existent
> on Linux)
>
> * periodic drops of sustained streaming bandwidth of as much a factor of 50, which can
> last for several seconds (e.g a disk that can peak at 100MB/sec fails to deliver
> better than 5MB/sec for a noticeable period).
>
> * a requirement to read larger blocksizes to get the same bandwidth than on other platforms
>
> Our test application is small, less than 130 lines of code. It uses the POSIX API to read a specified blocksize from each of N files, and reports on the observed I/O bandwidth. It comes with a companion shell script (about 60 lines) which sets up the files to be read and then runs the executable with each of series of (specified) blocksizes.
>
> I've attached both files (source code for the executable, plus the shell script). The executable has an unfortunate reliance right now on glib in order to get the cross-platform g_get_monotonic_time(). If you want to run them, the build command is at the top of the executable, and then you would just do:
>
> ./run-readtest.sh -d FOLDER-FOR-TEST-FILES -n NUMBER-OF-FILES -f FILESIZE-IN-BYTES blocksize1 blocksize2 ....
>
> (You'll also need some pretty large-ish chunk of diskspace available for the test files). The blocksize list is typically powers of 2 starting at 16348 and going up to 4MB. We only see problems with NUMBER-OF-FILES is on the order of 128 or above. To be a useful test, the filesizes need to be at least 10MB or so. The full test takes several (2-20) minutes depending on the overall speed of your disk.
>
> But you don't have to run them: you could just read the source code executable to see if any lightbulbs go off. Why would this code fail so badly once the number of files gets up to 100+? Why can we predictably make any version of OS X able to get barely 5MB/sec from a disk that can sustain 100MB/sec? Are there some tricks to making this sort of thing work on OS X that anyone is aware of? Keep in mind that this same code works reliably, solidly and predictably on Windows and Linux (and even works reliably on Windows as a guest OS on OS X).
>
> We do know, by simple inspection, that other DAWs, from Reaper to Logic, appear to have solved this problem. Alas, unlike Ardour, they don't generally share their code, so we have no idea whether they did something clever, or whether we are doing something stupid.
>
> I, and rest of the Ardour community, would be very grateful for any insights anyone can offer.
>
> thanks,
> --p
>
>

References:
	>Re: large scale (audio) file I/O on OS X : help or insight requested (From: Hamilton Feltman <email@hidden>)