Re: Unit testing a plug-in architecture
Re: Unit testing a plug-in architecture
- Subject: Re: Unit testing a plug-in architecture
- From: Derek Wyatt <email@hidden>
- Date: Fri, 15 Dec 2006 06:27:03 -0500
When it comes to proper unit testing, you simply want code that is
highly modular (as you say you have). For example:, code I've been
writing over the last month is designed to install a very complex and
highly dynamic environment (hundreds of possible "plug ins" that are
hundreds of megs each with thousands of files). One of the things I
need to have is a runtime graph of the installation components and
all of their dependencies. The entire system has 1500 automated test
assertions (for the entire system, not just the graph) and some of
those are:
1) create a graph component from a file in the filesystem: does it
have all of the right properties? (this is about 5-10 assertions)
2) create 2 components that have a dependency relationship: check
their properties and that their connection is solid (this is about 10
assertions)
3) load up a group of components with an invalid connection list
(i.e. a graph cycle): check to ensure the cycle is correctly detected
(this is 5 assertions)
4) load a group of components and select a leaf component: do all
of the dependency components get selected as well? (this is about 5
assertions)
etc etc...
They are very small tests that work on very small parts of the
system. And for the complex parts that require a ton of small pieces
(e.g. an actual installation test), we don't test all of the small
pieces in that go -- that would be insane -- we test the new
functionality given by the new complexity we've introduced and that's
all (in the installation case, were all of the appropriate programs
run, and did they run in the right order -- the graph is used very
heavily in installation but I've already tested that thoroughly so
it's not part of this test). If you add the next smallest amount of
complexity for testing, and it turns out that the smallest amount is
incredibly large (i.e. if I add this piece of this module, I need to
add 300 other modules, and that's the absolute smallest next step)
then you've made a design error and should probably refactor.
Regs,
Derek
On 14-Dec-06, at 12:53 PM, leenoori wrote:
El 14/12/2006, a las 15:25, Derek Wyatt escribió:
It's hard to say exactly, but in general one would decouple the
two by requiring there to be a test framework outside of the two
that is responsible for joining the logic (in the spirit of all of
the "unit" test frameworks out there, etc...).
Thanks for the input, Derek.
In other words, tests like this:
- For each plugin X do
- Using the application's plugin load framework, can we load
plugin X
- Using the application's state engine, can we fire the plugin
through it?
- etc etc..
The application doesn't have a framework for loading plug-ins, it
just has a class that is responsible for loading them. Are you
suggesting that this should be split out into a framework so that a
test bundle could directly access it? (Or equivalently, just link
against the app as BUNDLE_LOADER.)
So the test framework is outside of both systems and, if your
application framework is abstracted sufficiently then the amount
of reuse is in the 99% area so there is very little duplication in
the test framework. But if you need your entire app to test any
small feature then you're probably baked.
I dunno how you do it, but this is how I do it. It's really hard
to get an impression of what you're doing because I'm sure your
app is sufficiently complex to make any guess, well... stupid...
but it sounds to me like you've got a situation where your app is
lacking in sufficient separation to write modular tests and in
that case, you might have to pollute, or refactor.
Well, it's an interesting question. The whole point of the plug-in
architecture was that it would enable me to add optional stuff to
the app in a completely modular fashion. In this way, if a user
doesn't want a particular piece of functionality they don't even
have to dedicate memory to it and launch times are quicker. The
dependence is entirely one-way; that is, each plug-in depends on
the application and in fact specifies it as its BUNDLE_LOADER but
the application does not depend on any plug-in in any way and never
makes any reference to any code in any specific plug-in.
So in other words the application and the plug-ins themselves are
highly modular and I believe well compartmentalized, and I did it
this way because I believed it was the best design.
The issue is, how do I keep the unit testing side of things nice
and clean and elegant? I am not sure how I could do what you
suggest (get the test framework to do the joining)... that would
require the test framework to know about the internal details of
how the application loads the plug-ins, so I've just moved the
location of the "ugly pollution" from the application to the
framework, but the ugliness is still there. And I don't really want
to modify the testing framework itself anyway because that would be
adding project-specific stuff to a framework that's intended for
general-use.
I'm wondering if what you were getting at was the following kind of
set up between the components:
- Application (contains logic for loading plug-ins)
- Unit tests for application (in separate bundle)
- Plug-ins to be tested
- Unit tests for plug-in (separate bundle for each plug-in)
- "Glue" bundle that manages the running of the plug-in tests
That is, the "glue" bundle would link against the application using
BUNDLE_LOADER, and would therefore be able to use the application's
plug-in management code to get a list of loaded plug-ins, load plug-
in test bundles as necessary and fire off the tests. Seeing as
there is already one bundle being loaded for the application unit
tests, the "glue" may as well be rolled into that one. It's a
little bit nasty but it may work.
Anyway, I'll keep thinking about it and if you have any other ideas
let me know!
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Xcode-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden