• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Coreaudio-api Digest, Vol 12, Issue 43
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Coreaudio-api Digest, Vol 12, Issue 43

  • Subject: Re: Coreaudio-api Digest, Vol 12, Issue 43
  • From: Bibhas Acharya <email@hidden>
  • Date: Thu, 05 Mar 2015 09:10:30 +0545
I'm not sure this would be of any help, but perhaps you should look into UBC (http://wagerlabs.com/blog/2008/03/04/hacking-the-mac-osx-unified-buffer-cache/).

On Thu, Mar 5, 2015 at 1:45 AM, <email@hidden> wrote:
Send Coreaudio-api mailing list submissions to
        email@hidden

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.apple.com/mailman/listinfo/coreaudio-api
or, via email, send a message with subject or body 'help' to
        email@hidden

You can reach the person managing the list at
        email@hidden

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Coreaudio-api digest..."


Today's Topics:

   1. large scale (audio) file I/O on OS X : help or insight
      requested (Paul Davis)


----------------------------------------------------------------------

Message: 1
Date: Tue, 03 Mar 2015 18:20:31 -0600
From: Paul Davis <email@hidden>
To: coreaudio-api <email@hidden>
Subject: large scale (audio) file I/O on OS X : help or insight
        requested
Message-ID:
        <email@hidden>
Content-Type: text/plain; charset="utf-8"

Many readers of the list will know who I am, but for those who don't let me
start by just mentioning that I'm the lead developer and designer of
Ardour, an open source GPL'ed DAW for Linux, OS X and Windows, and also the
original author of JACK, a cross-platform API for low latency, realtime
audio and MIDI, including inter-application and network routing.

Over in Ardour-land, we've wrestled for a couple of years now with less
than ideal support for OS X and are just finally beginning to resolve
many/most of the areas where we were falling down.

But there's an area where we've really run into a brick wall, despite the
extremely deep and amazingly talented pool of people we have as developers,
people who know *nix operating systems inside out and upside down.

That area is: getting acceptable disk I/O bandwidth on OS X when reading
many files. For comparison: we can easily handle 1024 tracks on a single
spinning disk on Linux. We can get close to similar performance on Windows.
We can even (gasp!) get in the same ballpark on Windows running as a guest
OS on OS X (i.e. the entire Windows filesystem is just one file from OS X's
perspective).

But on OS X itself, we are unable to get consistent, reliable performance
when reading many files at once. Our code is, of course, entirely
cross-platform, and so we are not using any Apple specific APIs (or
Linux-specific or Windows-specific). The same code works well for recording
(writing) files (though OS X does still perform worse than other platforms).

We've used various dtrace-based tools to try to analyse what is going on:
these only confirmed for us that we have a problem, but provided no
insights into what.

Things became so inexplicable that we decided to break out the problem from
Ardour itself and write a small test application that would let us easily
collect data from many different systems. We've now tested on the order of
a half-dozen different OS  X systems, and they all show the same basic bad
behaviour:

     * heavy dependence of sustained streaming bandwidth on the number of
files being read
          (i.e. sustained streaming bandwidth is high when reading 10
files, but can be very low
                 when reading 128 files; This dependence is low on Windows
and non-existent
                 on Linux)

     * periodic drops of sustained streaming bandwidth of as much a factor
of 50, which can
          last for several seconds (e.g a disk that can peak at 100MB/sec
fails to deliver
          better than 5MB/sec for a noticeable period).

     * a requirement to read larger blocksizes to get the same bandwidth
than on other platforms

Our test application is small, less than 130 lines of code. It uses the
POSIX API to read a specified blocksize from each of N files, and reports
on the observed I/O bandwidth. It comes with a companion shell script
(about 60 lines) which sets up the files to be read and then runs the
executable with each of series of (specified) blocksizes.

I've attached both files (source code for the executable, plus the shell
script). The executable has an unfortunate reliance right now on glib in
order to get the cross-platform g_get_monotonic_time(). If you want to run
them, the build command is at the top of the executable, and then you would
just do:

  ./run-readtest.sh -d FOLDER-FOR-TEST-FILES -n NUMBER-OF-FILES -f
FILESIZE-IN-BYTES blocksize1 blocksize2 ....

(You'll also need some pretty large-ish chunk of diskspace available for
the test files). The blocksize list is typically powers of 2 starting at
16348 and going up to 4MB. We only see problems with NUMBER-OF-FILES is on
the order of 128 or above. To be a useful test, the filesizes need to be at
least 10MB or so. The full test takes several (2-20) minutes depending on
the overall speed of your disk.

But you don't have to run them: you could just read the source code
executable to see if any lightbulbs go off. Why would this code fail so
badly once the number of files gets up to 100+?  Why can we predictably
make any version of OS X able to get barely 5MB/sec from a disk that can
sustain 100MB/sec? Are there some tricks to making this sort of thing work
on OS X that anyone is aware of? Keep in mind that this same code works
reliably, solidly and predictably on Windows and Linux (and even works
reliably on Windows as a guest OS on OS X).

We do know, by simple inspection, that other DAWs, from Reaper to Logic,
appear to have solved this problem. Alas, unlike Ardour, they don't
generally share their code, so we have no idea whether they did something
clever, or whether we are doing something stupid.

I, and rest of the Ardour community, would be very grateful for any
insights anyone can offer.

thanks,
--p
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.apple.com/mailman/private/coreaudio-api/attachments/20150303/ec2826b2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: readtest.c
Type: text/x-csrc
Size: 4120 bytes
Desc: not available
URL: <https://lists.apple.com/mailman/private/coreaudio-api/attachments/20150303/ec2826b2/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run-readtest.sh
Type: application/x-sh
Size: 1737 bytes
Desc: not available
URL: <https://lists.apple.com/mailman/private/coreaudio-api/attachments/20150303/ec2826b2/attachment.sh>

------------------------------

_______________________________________________
Coreaudio-api mailing list
email@hidden
https://lists.apple.com/mailman/listinfo/coreaudio-api

End of Coreaudio-api Digest, Vol 12, Issue 43
*********************************************

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden
  • Follow-Ups:
    • Re: Coreaudio-api Digest, Vol 12, Issue 43
      • From: Paul Davis <email@hidden>
  • Prev by Date: rendered sound is crackly under iOS8
  • Next by Date: Re: Coreaudio-api Digest, Vol 12, Issue 43
  • Previous by thread: rendered sound is crackly under iOS8
  • Next by thread: Re: Coreaudio-api Digest, Vol 12, Issue 43
  • Index(es):
    • Date
    • Thread