Re: Audio Units and virtual device implementation
Re: Audio Units and virtual device implementation
- Subject: Re: Audio Units and virtual device implementation
- From: Colin Martel <email@hidden>
- Date: Mon, 15 Jun 2009 18:05:57 -0400
Jeff, Brian,
Sorry, having worked on this music research project for over a month,
I tend to forget how different this is from standard industry audio
work. :) I should probably clarify with a bit of context.
The program is (for now) mostly used for experimental/artistic
electroacoustic compositions with a spatial component. The studio we
have set up at Montreal University has 36 speakers laid out in a dome.
Since mapping spatial compositions to speakers one by one would be
rather tedious, Zirkonium instead uses VBAP (Vector-Based Amplitude
Panning, described in an academic paper available on Google), which
basically involves an algorithm to convert a single unit vector into
panning information within the dome.
This vector is what's controlled by the audio unit, nothing more. In a
way, calling it an *Audio* unit is a bit of a misnomer, as it doesn't
do any signal processing but only presents some parameter automation
for the panning vector. If the SDK allowed for it, the AU could even
give { {1, 0}, {2, 0} } as its supported channels, since it doesn't
actually output anything.
Now, since a typical piece will have many different tracks often
moving around independently throughout its duration, each of these has
to be passed to Zirkonium separately in order for it to compute the
signal coming from each speaker within the dome. This is where the
virtual device (as a form of HAL plug-in, yes) comes in. It provides
the host program a number (currently fixed to an arbitrary 32) of
output sources, and I think this is the misleading part, as the
outputs to the virtual device do NOT represent actual speakers, but
rather individual sources that can be controlled by adjusting their
vectors.
Now, in order to match the timeline of the piece, vector control has
to be done within the host program -- with the aforementioned Audio
Unit parameter automation. This information is sent to the Zirkonium
process, in parallel with the audio stream using a CFMessagePort.
The problem I was trying to describe is that the Audio Unit has to be
told manually which of the virtual device's outputs it is controlling
each time you instantiate it, something which can be error-prone given
how automation counts the channel as a writable parameter. Even while
aware of that potential mistake, identifying mismatches between the
device parameter and the audio unit can range from confusing to
downright frustrating with host programs that allow arbitrarily
mapping stereo channels to outputs, such as DP. Ergo, making the Audio
Unit automagically figure out what channel number it should be
controlling would make the whole thing that much simpler.
Re: Brian
> I also did not follow your distinction between 'simulated' panning and
> 'actual' panning. You'll have to define what you mean by those terms.
That was a bit of a sidestep. I meant that as far as I can tell,
Apple's example aupn units provided (AUSoundFieldPanner, HRTFPanner,
AUVectorPanner, AUSphericalHeadPanner) reproduce sound in space over a
limited amount of speakers by using DSP to alter the sound (with
doppler effects, reverb, etc.) as it would over distance and space. In
a way, Zirkonium is doing the contrary -- creating spatial
representation of a few sources with an actual large number of
speakers.
Hope that clears things up. It's a very niche situation. :)
-cmartel
P.S.: Perhaps I'm mistaken about the example panners -- like I
mentioned, there's very little documentation to be found on aupn
units.
On Mon, Jun 15, 2009 at 4:22 PM, Jeff Moore<email@hidden> wrote:
> I don't think I understand entirely what you are describing here. At least
> not when put together like this. I'll try my best though =)
>
> At any rate, I get the part about using a fake audio device (presumably
> implemented via a HAL plug-in) to pipe audio to your app's process from a
> target process. It sounds to me like this part is indispensable presuming
> that all the interesting spatialization processing is happening in your
> app's process.
>
> I also get the part about using an AudioUnit in the process using the fake
> device to send control information to your application about what to do with
> the audio. The AU provides parameters for the host to automate to control
> the spatialization processing being done in your app, including a parameter
> that says what channel on the fake device the AU is controlling.
>
> I'm inferring that the reason for all of this is that you are ultimately
> providing a mapping from an N channel surround format to an M channel
> speaker array. The fake device acts as the N channel audio device and sends
> the audio to your app which does the processing and then sends the resulting
> M channels to the actual hardware.
>
> My first thought about this is that it seems like a pretty convoluted
> solution to something that could be done entirely in the fake device.
>
> Basically, you'd just drop all your spatialization processing into the
> N-channel fake device and then drive the fake device using the M-channel
> real device. Then, your AU could be simplified down to one that could tell
> your fake device which real device to use for IO and to allow for automating
> all the parameters.
>
> This solution simplifies the signal path which ought increase performance
> and lower latency while still affording the integration you are looking for.
>
> That said, I'm not an AU expert. So I've have glossed over a lot of details
> on the AU side of things. But merging your N to M processing into the fake
> audio device seems like a big win to me - presuming my understanding of the
> problem is correct.
>
>
>
> On Jun 15, 2009, at 9:21 AM, Colin Martel wrote:
>
>> Hello,
>>
>> I'm working on Zirkonium, a program that has a bit of a particular
>> setup. To put it briefly, it handles spatialization of multiple (>8)
>> channel audio for speakers placed in a dome setup. In order to
>> collaborate with host programs, many of which I'm told don't support
>> panning over a large number of speakers, it creates a virtual device
>> with an arbitrary number of channels which the host program can
>> select. Each channel in the device basically represents an entity
>> which can then panned by Zirkonium over the N amounts of speakers
>> present in the dome configuration.
>>
>> In order to be able to change and automate that panning from within
>> the host program, an Audio Unit is used which sends the angles and
>> whatnot to Zirkonium. However, since the audio unit is a separate
>> piece of code from the virtual device, I'm running into some
>> difficulty finding an easy way to make sure the audio unit is panning
>> the channel it's attached to. The current setup is to select the
>> virtual device channel in the host program, then match that channel in
>> the audio unit. But since the channel becomes an actual AU parameter,
>> it becomes subject to automation, can be mismatched from the device
>> channel and the whole thing is rather confusing and error-prone.
>>
>> I'm trying to find an alternative, but since I'm very new to
>> CoreAudio, I'm unsure what can and cannot work. Here are the ideas
>> I've come up with:
>>
>> 1. Have the audio unit extract the device channel number from the
>> stream. This would be optimal but impossible as far as I can see, as
>> it would basically imply making low-level details visible to
>> AudioUnits that would break their ability to be used independently.
>> Maybe this could be done by bypassing the AU SDK? Maybe it does
>> provides a way to access device info and I just didn't see it?
>>
>> 2. Abandon the virtual device implementation and instead pass the
>> audio frames through the AudioUnit along with the spacialization info.
>> The AudioUnit would then choose the proper device channel to match the
>> settings. This sounds like an aupn unit, however the examples from
>> Apple that I've seen all seemed to involved simulating panning over
>> stereo through different DSP algorithms, as opposed to actually
>> panning. Documentation on aupn units is scarce.
>>
>> 3. Abandon the virtual device implementation and pass the audio frames
>> through the AudioUnit in a custom stream format, basically bypassing
>> the HAL. Based on what I can see, this is the only solution that would
>> work for sure, but sending the information in this custom way sounds
>> like it might incur latency. But since the virtual device is already a
>> software solution regardless of using the HAL or not, perhaps it
>> wouldn't be so bad?
>>
>> I realize this is probably a bit outside of what AudioUnits are
>> supposed to do, but the original programmer tells me host programs
>> rarely handle speaker configurations with more speakers than the
>> standard surround setups, hence the external program requirement. If I
>> could at least make that interaction simpler, then it'd add a lot of
>> value to the AudioUnit setup.
>>
>> Sorry for the lengthy explanations, but I think this is a bit outside
>> standard fare so I want everything to be clear. :)
>>
>> -cmartel
>
>
> --
>
> Jeff Moore
> Core Audio
> Apple
>
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Coreaudio-api mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden