Re: Real-time buffer playback in Swift?
Re: Real-time buffer playback in Swift?
- Subject: Re: Real-time buffer playback in Swift?
- From: Zack Morris <email@hidden>
- Date: Wed, 17 Sep 2014 09:46:17 -0600
On Sep 16, 2014, at 1:23 PM, Ian Kemmish <email@hidden> wrote:
>
> On 16 Sep 2014, at 14:56 -0400, Rob Nikander <email@hidden> wrote:
>
>> On Tue, Sep 16, 2014 at 8:10 AM, Paul Davis <email@hidden>
>> wrote:
>>>
>>>
>>> I'm puzzled why you think this is possible. Last time I read about this,
>>> Apple's position was that realtime (i.e. low latency) audio still required
>>> C/C++.
>>>
>>
>> Do you remember where you read that? I've not seen such a statement, and I
>> too was just wondering if I could write an AudioUnit render callback in
>> Swift.
>
> An AudioUnit render callback is an interrupt handler, to all practical intents and purposes. To risk typing a generalisation on the Internet: If the language you're using lets you write interrupt handlers, then you can write a render callback in it; if it doesn't, you can't.
>
> Does Swift let you do that? If, as stated earlier in this thread it doesn't let you take the address of a function (because that's unsafe), then clearly it cannot.
>
> In addition, Apple's propaganda for Swift suggests that it's a just-in-time compiled language, which might mean trying to run the compiler the first time your callback gets called back, which in turn would require allocating memory, which in turn breaks the commandment that thou shalt not do anything which might conceivably block during real-time code.
>
>
> The easiest way out of this design decision you've taken would seem to be to write all of the real-time code in C and then either i) find a way of importing an arbitrary C library into a Swift project, or ii) put it all in a separate server process and have your Swift code talk to your C code via your favourite kind of IPC. (Back when the world was young I wrote graphics and window bindings for various interpreted languages like APL, Forth and Lisp, and I had to adopt both of these approaches in different circumstances.)
I’ve been thinking about this a lot lately. With all of the new languages like Go, Rust, Swift, Node, Haskell, etc, we’re seeing a lot of ideas about how processing can be made simpler, more intuitive, more readable, more expressive, and give the programmer more leverage and greater productivity. But we’re stuck with some of the old conventions like interrupts and shared memory that are powerful performance-wise but throw a wrench in purity (don’t get me wrong, I grew up on interrupts, atomic functions and OpenTransport and still have a certain twisted fondness for them... I just shudder to think about trying to use them today).
Reading about Minix got me thinking about the idea of adding userland hooks in a program to communicate with low-level kernel code:
http://www.minix3.org/330.html
Realtime processing in CoreAudio is akin to something running at kernel level because it has to be realtime. But the stream processing itself should really be using buffers where language and priority (within reason) don’t matter. I know it’s easy to worry about nondeterministic lags during compilation and garbage collection, but some of the other lags in non-realtime systems are holdovers from when kernels just weren’t very good. They made tradeoffs in overhead to keep throughput high at the cost of latency. The symptom of this today is that computers are 1000 times faster than a few decades ago, but sleep times haven’t shrunk by a factor of 1000. So if we aren’t super careful, realtime stuff like audio cuts out occasionally even though it’s processing a paltry 44100 samples of 4 bytes each (roughly 175 kB) per second. The i5 processor in my Mac Mini can write 10 GB/sec:
http://www.techspot.com/review/193-intel-core-i5-750/page5.html
So the memory is roughly 50,000 times faster than what’s needed. If we imagine each sample taking a clock cycle, a 2.5 GHz CPU is still about 15,000 times faster than needed. That means that even if we use the slowest interpreted language, running 200 times slower than C or more, the computer is still on the order of 100 times faster than we need.
I realize there are some issues here with decompressing formats like mp3 that take orders of magnitude more processing, but if I remember right in the 90s, decoding an mp3 only took about 5% processor on a 100 MHz CPU. My computer is 25 times faster than that, so even at 0.2% CPU, that’s a safety margin of 500, which means it could process TWO mp3 streams in an interpreted language. Well, technically 4 or 8, depending on whether the other core was used and hyper-threading. That’s not even considering that most interpreted languages recruit system calls which are usually written in C or have hardware acceleration. I’m talking decoding mp3 by hand in a scripting language with no compilation or optimization, and still being able to play several at once. It’s easy to forget just how fast computers have gotten.
So the main problem seems to be stop-the-world garbage collection. Generally garbage collection is implemented pretty poorly. For example, they make assumptions like: the process will be long-lived, memory constraints will be tight, all of the threads have to be stopped in case they share memory, etc. Small, one-off processes like the kind run in Apache likely don't need garbage collection. So why can’t functions in high level languages work more like goroutines (from Go), except running in their own isolated processes, and avoid most of these issues? Combining APIs like Objective-C, Swift and C would look more like shell programming, where you spawn a process in some language, it hangs around in the background, and you pass data to it with pipes.
Without digressing too much, I just think that it would be a great if inter-language communication was a standardized API, so for example code might look like this:
typical single threaded code (Objective-C, Node, etc)...
run<specification, language, inputs, outputs, memory pool, shared memory locations, function pointers, etc>
{
compile just in time
loop
{
read inputs
process data, spawn other processes, allocate memory, etc
write outputs
garbage collect
}
}
typical single threaded code (Objective-C, Node, etc)...
The run block is just using a convention similar to what C uses for inline assembly with the asm keyword:
http://en.wikipedia.org/wiki/Inline_assembler
It would handle munging data in a standardized way to work between languages. So in this case, the outer code file would be written in Swift, the inner run block would be written in Swift, and the run layer between them that references the CoreAudio function pointers would be written in C or assembly and conform to the run spec. The run spec would probably look like Rust, where the details are spelled out exactly. So the spec for a particular CoreAudio callback might look like this pseudo code:
run<spec:CoreAudio::some_callback, language:Swift, inputs:inPipes, outputs:outPipes, ...>
{
}
The other Swift process would get spawned waiting for the CoreAudio callback, and the only overhead would be in context switching and copying data over the pipes (optimized with something like copy-on-write).
Maybe it could only mix 100 streams instead of 1000 or whatever, but I think the gains in programmer productivity would be worth it. Also stream processing is computationally pretty simple, because CPUs today mostly sit around waiting for RAM. I would expect Java and C to be able to handle the same number of streams. I might even extend that as far as optimized Javascript:
http://asmjs.org/faq.html
Python, Perl, Lua, etc would be only a little slower.
Language today really shouldn’t matter. But it does, so I think that points to smells in the OS that should have been addressed years ago. I’m sure any Swift architects reading this are rolling their eyes right now, but if they want to make programming a better place, they could go ahead and write this run layer thingy. Ya... that would be great.
Zack Morris
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden