Re: otest -- only useful for bundles?
Re: otest -- only useful for bundles?
- Subject: Re: otest -- only useful for bundles?
- From: James Bucanek <email@hidden>
- Date: Sun, 6 Nov 2005 17:49:49 -0700
Chris Hanson wrote on Saturday, November 5, 2005:
>Ah. When running dependent unit tests for a command-line tool or
>application, otest isn't actually involved. It's only used when
>you're running unit tests for a framework, since something needs to
>actually load the framework in order to test it.
Gotcha. After spending considerably more time with the RunUnitTest script, I'm beginning to see what's going on.
>Ah. Does your PrimesToolTest target actually have any sources
>associated with it? And does the name of the generated executable
>match the CFBundleExecutable key in the test bundle's Info.plist file?
Bingo! I had renamed the target and thought I had fixed all references to the executable name, but missed the CFBundleExcutable key. Thanks. My unit tests on my command-line tool are now working. (yea!)
Since I'm trying to learn enough of this to explain it to others, let me summarize what I've absorbed to date. (I still have some questions, which I'll get to in a moment.)
- When building an "independent" test suite, the source for both the tests and the code being tested are compiled and linked together into a single executable bundle which is the product of the unit test target. The resulting bundle (test code and application code) is loaded and executed by the otest tool.
- When testing a framework or dynamic library, only the code for the tests are compiled into the unit test bundle. The otest is executed to load both the unit test bundle and the framework to be tested and execute the tests.
- When testing an application, the code for the tests are compiled into the unit test bundle. When it come times to execute the tests, special DYLD_... environment variables are set so that the dynamic linker links the target executable to special unit test frameworks and the unit test bundle (which contains the actual tests). When the application begins executing, the "injected" bundle/framework intercepts the application's execution, runs the tests, and exits.
Here are my questions:
-- Question #1 --
In the documentation, a comment to effect of "if you want to run just a subset of the unit tests, you need to rewrite the custom script phase of the target." This is, I fear, a gross understatement. After having played with various scripts, I came up with the following. This script assumes that you are debugging an application and have TEST_HOST defined. A different script would need to be developed for independent tests and/or frameworks.
The script looks for a build setting of TEST_SET. If undefined or set to "All", the script runs the normal RunUnitTests script. Otherwise, it emulates the setup performed by RunUnitTests and launches the target application with a -SenTest argument of ${TEST_SET}. To run a specific test, one sets the TEST_SET build setting to "Class/Test" as desired. To run all of the tests, simply delete the build setting or set it to "All".
----------------------------------------------------------------------
# TEST_SET must be All or the subset of tests to run
if [ "X${TEST_SET}" = "X" ]; then
TEST_SET=All
fi
### Run all of the unit tests in this bundle.
if [ "${TEST_SET}" = "All" ]; then
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
exit
fi
### Run a specific subset of tests
# To run a subset of tests, we have to call the otest tool directly,
# first setting up all of the prerequisite environment variables.
if [ "X${TEST_BUNDLE_PATH}" = "X" ]; then
TEST_BUNDLE_PATH="${BUILT_PRODUCTS_DIR}/${TARGET_NAME}.${WRAPPER_EXTENSION}"
fi
# Set and export the dynamic framework variarbles
if [ "X${DYLD_FRAMEWORK_PATH}" != "X" ]; then
DYLD_FRAMEWORK_PATH="${BUILT_PRODUCTS_DIR}:${DYLD_FRAMEWORK_PATH}"
else
DYLD_FRAMEWORK_PATH="${BUILT_PRODUCTS_DIR}"
fi
if [ "X${DYLD_LIBRARY_PATH}" != "X" ]; then
DYLD_LIBRARY_PATH="${BUILT_PRODUCTS_DIR}:${DYLD_LIBRARY_PATH}"
else
DYLD_LIBRARY_PATH="${BUILT_PRODUCTS_DIR}"
fi
export DYLD_FRAMEWORK_PATH DYLD_LIBRARY_PATH
export DYLD_NEW_LOCAL_SHARED_REGIONS=YES
export DYLD_NO_FIX_PREBINDING=YES
if [ "X${DYLD_INSERT_LIBRARIES}" != "X" ]; then
DYLD_INSERT_LIBRARIES="${SYSTEM_LIBRARY_DIR}/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection:${DYLD_INSERT_LIBRARIES}"
else
DYLD_INSERT_LIBRARIES="${SYSTEM_LIBRARY_DIR}/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection"
fi
export DYLD_INSERT_LIBRARIES
# Set the bundle path for C/C++ applications
XCInjectBundle="${TEST_BUNDLE_PATH}"
export XCInjectBundle
# Run the application using the special dynamic libraries passing the tests to be performed
"${TEST_HOST}" -SenTest "${TEST_SET}"
unset DYLD_INSERT_LIBRARIES
unset XCInjectBundle
----------------------------------------------------------------------
This script appears to work for all of the projects I've tried so far. If anyone sees anything amiss, I would appreciate it if you would point out the problem.
I also feel that this is something that really should be part of the RunUnitTests script. It would take only minor modifications to RunUnitTests for it to look for a build setting like TEST_SET and adjust its launch arguments accordingly, rather than force the developer to reverse-engineer the script and duplicate all of the required set-up and initialization that the unit test framework requires. Especially since those requirements don't seem to be documented anywhere.
If anyone else agrees with me, I'll be happy to post an enhancement request.
-- Quiestion #2 --
I've got this working for an Objective-C command-line utility and a Cocoa application, but it still fails to run the tests for my C++ command-line tool. I double checked everything I did for the Obj-C test case, and I appear to have everything set up correctly -- according to the instructions at least.
- I created a unit test target set up just like the Obj-C test target.
- ZeroLink is off for the application target.
- TEST_HOST & BUNDLE_LOADER are set to the C++ executable
- Wrote some C++ unit tests and added them to the unit test target.
- I added the static test objects to the source code of the unit tests:
SieveOfEratosthenesTests sieveTestPrimes(TEST_INVOCATION(SieveOfEratosthenesTests,testPrimes));
SieveOfEratosthenesTests sieveTestNonPrimes(TEST_INVOCATION(SieveOfEratosthenesTests,testNonPrimes));
SieveOfEratosthenesTests sieveTestMapEdges(TEST_INVOCATION(SieveOfEratosthenesTests,testMapEdges));
...
- I added the following code to my main():
TestRun run;
// Create a log for writing out test results
TestLog log(std::cerr);
run.addObserver(&log);
// Get all registered tests and run them.
TestSuite& allTests = TestSuite::allTests();
allTests.run(run);
// Log a final message.
std::cerr << "Ran " << run.runCount() << " tests, " << run.failureCount() << " failed." << std::endl;
return (0);
- I had to link my main application to the CPlusTest.framework to get it to link.
Here's where it gets stuck. The application builds OK, but then Xcode tries to build the PrimesToolTest bundle, and I get a link error.
/usr/bin/ld: Undefined symbols:
SieveOfEratosthenes::isPrimeInMap(int)
SieveOfEratosthenes::isPrime(int)
SieveOfEratosthenes::SieveOfEratosthenes(int)
SieveOfEratosthenes::~SieveOfEratosthenes()
collect2: ld returned 1 exit status
This, apparently, prevents the test bundle from being created and the test fails when it tries to run:
/bin/sh -c /Users/james/Desktop/PrimesCpp/build/PrimesCpp.build/Debug/PrimesToolTest.build/Script-1AEE2AD7091ECA5500841C5A.sh
2005-11-06 17:14:09.844 PrimesCpp[13795] CFLog (21): Cannot find executable for CFBundle 0x300dd0 </Users/james/Desktop/PrimesCpp/build/Debug/PrimesToolTest.cptest> (not loaded)
DevToolsBundleInjection: Error loading bundle '/Users/james/Desktop/PrimesCpp/build/Debug/PrimesToolTest.cptest'
Suite All started
Suite All finished
Ran 0 tests, 0 failed.
This is a "dependent" test suite, so the source code to the classes I want to test should not (according to the documentation) be included in the target. Am I missing some magic linker setting that allows the bundle to compile and link even though the classes that the tests reference are not present?
-- Question #3 --
Does the code I inserted into main() to run the test automatically turn off when the application is executed normally? Will it not spit out those annoying messages? How does my app know that tests should be run vs. when it should execute normally? Both the example code and the code I have above *alway* run the tests and exit. As its written now, my application won't ever run.
-- Question #4 --
When configuring a dependent unit test to test a framework or dynamic library, how should the target be configured? How does otest know which frameworks and libraries are being tested?
-- Question #5 --
Can you configure a dependent unit test to test a static library?
--
James Bucanek
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Xcode-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden