Re: Unit testing and Xcode integration
Re: Unit testing and Xcode integration
- Subject: Re: Unit testing and Xcode integration
- From: Nir Soffer <email@hidden>
- Date: Wed, 16 Aug 2006 02:40:24 +0300
On Aug 14, 2006, at 9:00 AM, Chris Hanson wrote:
I don't intend for my responses to be perceived as defensive; I
tend to provide a lot of information as background that can
sometimes look more like rationale. I genuinely want to know what
would make the experience of OCUnit within Xcode better from your
-- and others'! -- perspective.
Here is my perspective:
- Clicking on an error in Xcode build results does not always show
the relevant test, sometimes it just does nothing (XCode 2.1-2.3)
- When you see test failures, the failure description does not wrap
to the next line, but instead cut. If you wait a little, you see the
full text in a tooltip, but it is hidden before you can read the
whole text. So the only reliable way to see the failure is open the
build console, move the split down, because it hides your results
pane and then closing it, moving the split up to see more code. You
have to repeat this annoying ritual every error.
- It does not distinguish between failures and errors. An error is
unexpected condition while testing, a failure is when your assert
failed. SUnit does.
- Failure does not stop the test. For example:
- (void)testFail
{
id obj = nil;
STFail(@"next line should NOT run");
// some code that should NOT run if we failed above
STAssertNotNil(obj, nil);
}
When you run this, you get 2 failures instead of one, and the code
that should not run does run.
- Does not display test results in the results window, only in the
build console, which is pain to use, see above
- No skip support - sometime you want to skip some test, for example
when the test does not make sense (missing support library, different
platform etc.) - SUnit does.
- Inconvenient API - adding nil description everywhere - it MUST be
optional. I guess it is because of using macros, see bellow.
- Using ugly C macros instead of nice ObjC like ObjcUnit uses:
[self assert:result equals:expected];
[self assert:result equals:expected message:@"description..."];
- Test may crash the testing tool, which does not let you see other
tests until you fix the crash. I guess it will be hard to change
this, but it is very annoying
- Bogus results because of the integration with build:
- "Running custom shell script" - should be: "Running test suite"
- "error: -[Class test]: description" - should be "Failure: -
[Class test]: description"
- "Build failed" - should be "Tests failed"
- The status line: "Executed 354 tests, with 0 failures (0
unexpected) in 3.610 (4.389) seconds" is not displayed in the results
pane, only in the console.
- Inconsistent terms - "failures" in the console, "errors" in Xcode
build results.
- Test Case prefix in the console output make it harder to see your
failing test name. Of course its a test case
- Confusing test results in the console
Test Case '-[Class testA]' passed (0.000 seconds).
/Very/Long/Path/To/The/File/Being/Tested.m:80: error: -[Class
test] : next line should not run
Test Case '-[Class testB]' failed (0.006 seconds).
- The long path make it harder to see the test name
- The description should be after the test result, not before. It
looks as if the error description belong to testA while it describe
testB
It should be formatted like (removing the repeated "Test Case"):
[Class testName]: passed
[Class testName]: failed: description
[Class testName]: passed
Or:
[Class testName]: passed
[Class testName]: failed
description
[Class testName]: passed
- Lot of junk in the console output - because of the integration with
build instead of run. You my want to use the console to see all the
missing information in the build results pane, but then you have to
cope with this:
Building target “Tests” of project “Foo” with configuration “Debug”
Checking Dependencies
CompileC /blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/
blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/
blah/blah/blah/blah
... lot of these lines, none relevant to the tests
Ld /blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/
blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/
blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/
blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah/blah
PhaseScriptExecution /blah/blah/blah/blah/blah/blah/blah/blah/blah/
blah.build/Debug/Tests.build/Script-48AF29AC0A2C447B0009F916.sh
cd /blah/blah/blah/blah/blah/blah/blah/blah/blah/
setenv blah blah
... more irrelevant setenv lines
... more setenv lines
... even more
... where are my test results?
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... scroll
... I guess the point is clear now :-)
/bin/sh -c /blah/blah/blah/blah/blah/blah/blah/blah/blah.build/
Debug/Tests.build/Script-48AF29AC0A2C447B0009F916.sh
- Random order of test suites. I expect to see everything sorted in
the same way in all runs. It makes the output much easier to read
when you know where to look for results.
- Empty test suites run. If you try to avoid duplicate helper code by
using super class with helper methods, and some sub classes for the
actual tests, your super class will run 0 tests instead of just ignored.
- Super test method run in subclasses - If you try to avoid duplicate
code by creating a test subclass, the super test methods will run in
the subclass, instead of only the subclass methods.
- Using deprecated API, (gives the mature feel :-) ):
2006-08-16 00:33:55.040 otest[15165] +[NSATSGlyphGenerator
initialize] invocation. The class is deprecated.
Best Regards,
Nir Soffer
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden