Re: auval and you
Re: auval and you
- Subject: Re: auval and you
- From: "Angus F. Hewlett" <email@hidden>
- Date: Wed, 21 Mar 2007 10:31:55 +0000
Given that auval is an API validator, not a full boundschecker / purify
- style error detector, and performs virtually no validation at all on
the UI and interaction code where most such crash-bugs live, I consider
it very likely that users are living with that much of the time anyway,
auval or no :o/
Seems to me that the way this is being handled has two side-effects:-
- it effectively forces a refresh cycle on to third party developers
that these developers do not want
- it creates a concrete association in the user's mind between AU API
implementation validity, and overall code stability
The problem with this is that, the moment an AU fails validation, it
will then immediately get the blame for any crashes on the user's DAW --
which is misleading as it's trivially easy to write an AU which will
pass validation and then crash, or fail on a technicality and yet not
contain any of the kind of bugs that tend to lead to crashing. So if A's
plug fails validation, but B's plugin (or, horror of horrors, Logic
itself) is crashing the system, A will always get the blame.
So even if Apple isn't forcing developers to make AUs pass validation,
the users do so out of a misguided association between
implementation-validity and overall code stability. Also, many users
still don't even know about the AU Validation control panel in Logic,
and don't have much understanding of it.
I don't have a problem with this if it operates over a long (2 or 3
year) cycle, that fits in with the sort of time frame over which
platform and OS changes tend to render plug-ins unreliable or inoperable
anyways, and by that time there's usually a decent alternative to
anything that's no longer being developed. However, it seems to me that
placing this much emphasis on an automatic validator such as auval is,
at best, misleading the user base.
Best,
Angus.
William Stewart wrote:
Our customers do not give us the luxury of waiting for 2 to 3 years to
turn around critical bugs. I really am surprised that you consider it
reasonable to expect users to live with plugins that might be
scribbling over memory and causing random crashes and so forth for
such a period of time.
Regardless of this (and I think we'll have to agree to disagree here),
bare in mind that auval doesn't prohibit anything. For instance, in
Logic if an AU fails validation, the user can still bypass this and
use the AU; it still will behave exactly the same way as it did
before. We don't explicitly disable the AU completely (nor do we have
any plans to do that either). The difference is that the user now
knows that if they are experiencing instabilities in their projects,
then it could be due to problems with some of the audio units they are
using, rather than the host apps. Remember, all that the user knows is
that Logic crashed.
Finally, I would also reiterate that for many of these enhancements we
are giving you 2 to 3 years to fix them. As I said below, many of the
tests that now fail an audio unit in Leopard have been explicitly
bypassed for the 10.4.9 update for just the reason you ascribe (even
though some of these I consider to be quite serious problems). The
time difference between Tiger and Leopard is about 2 years. Ideally, I
would hope that the AUs themselves would be fixed before we explicitly
start failing them.
Bill
On 20/03/2007, at 12:54 PM, Angus F. Hewlett wrote:
Bill,
May I respectfully suggest that six months is NOWHERE NEAR LONG
ENOUGH when you're talking about breaking previously-'compatible'
software.
Yes, I realise you're keen to get new features rolled out, and have
host developers being in a position to take advantage of new features
with full confidence, but a six month window bears little relation to
a lot of devs' working cycles.
I'm sure we all have our own ideas as to what the cutoff period or
compatibility window should be, my £0.011 is that actual
compatibility should not be broken for perhaps 2 or 3 years after
full warning is given to developers.
There are a lot of AUs written in the last 2-3 years that are still
'current' from a user point of view, but may well be no longer
supported by their developer, or are just not at a phase in the dev
cycle where the dev is able to put out new releases at a few months'
notice. Are the gains from tightening the spec at this point really
big enough to cause end-users as much inconvenience as this may do?
Best regards,
Angus.
William Stewart wrote:
As many of you know, audio units are plugins that run within the
process (address space) of a host application. They enhance the
capabilities of the host apps, providing many choices to users for
synthesis, sampling, audio processing and so forth.
From the perspective of a host app, there are two primary conditions
that must be met:
(1) The API as specified for the audio unit must be implemented
correctly
(2) The behaviour of the audio unit must be predictable and reliable.
If these two criteria are not met the user can experience
instabilities when using the audio unit within a host application
(such as Logic, Garage Band, Digital Performer, etc), including
catastrophic consequences such as crashing and the loss of unsaved
data.
auval is a tool that was designed to validate the basic behaviour
and operation of an audio unit. Its primary focus applies to two areas:
(1) API conformance/correctness
- does the audio unit implement the semantics of the API
contract correctly?
(2) Behaviour
- does the audio unit have bugs that would cause random crashes
or other undesired consequences?
An audio unit passed through the validation process can have several
results:
(1) PASS
- the API is implemented correctly and no problems have been
detected
(2) PASS with warnings
- the audio unit may have some problems in some situations, but
in general it is behaving correctly
In the following two cases the audio unit should be considered to be
unreliable and problematic if used:
(3) FAIL with errors
- the audio unit exhibits some significant problems that would
impact its general usability
(4) Crashes
- in some cases the auval tool is not able to directly detect a
problem with the audio unit, but it can cause something bad to
happen in the audio unit through the testing that it does.
Regardless of any of these results, auval has no actual influence or
affect on the audio unit when it is used within a host app.
The primary API specification for audio units was described with Mac
OS X 10.2; it is known colloquially as "Audio Unit v2"; version two
of the API. The API calls are expressed primarily in the
AudioUnit.framework, in the header files AUComponent.h and
AudioUnitProperties.h. Audio units have of course developed since
then, with new features added to accommodate different scenarios and
user requirements, however the basic API semantic and compatibility
has remained as defined at that time.
The auval tool has also developed since then; it has changed to test
and validate new features and is also able to detect problems now
that it previously wasn't able to. We regularly seed copies of auval
to all audio unit developers to ensure that the auval tool is not
itself introducing problems as well as to provide a chance for
feedback and other comments. We seed as often as we can; providing
builds months ahead of when we would expect these to be released to
the general public. We expect developers, just as our users expect
this of us, to incorporate fixes to these problems in their updates,
hopefully before a new version auval itself is released.
As an example of this, the changes that were made to auval as
released in 10.4.9 were largely made and seeded to developers in
Sept/Oct 2006 (some of these changes even earlier than this). We
also disabled some of the tests as they would apply to a Tiger based
system because we didn't think they warranted a failure on such a
system. In these cases, they will however fail when running against
the auval installed with a Leopard system; this version has also
been seeded to developers.
If you are not on the auval seeding program and you are developing
audio units, then please contact me (William Stewart: email@hidden).
We'll gladly answer any comments or concerns.
Thanks
Bill
--mailto:email@hidden
tel: +1 408 974 4056
__________________________________________________________________________
"Much human ingenuity has gone into finding the ultimate Before.
The current state of knowledge can be summarized thus:
In the beginning, there was nothing, which exploded" - Terry Pratchett
__________________________________________________________________________
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--mailto:email@hidden
tel: +1 408 974 4056
__________________________________________________________________________
"Much human ingenuity has gone into finding the ultimate Before.
The current state of knowledge can be summarized thus:
In the beginning, there was nothing, which exploded" - Terry Pratchett
__________________________________________________________________________
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Coreaudio-api mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden