Camino Tests

There are two levels of testing, unit tests for individual classes and methods inside the code, and application tests that are calls to the Camino commands. All of these tests live in the camino/test directory. We can always use more tests. New code should always come with tests. If a bug is found in existing code, add a test to catch the bug before fixing it - that way we can ensure that bugs never return.

Unit tests

The unit tests are written in the JUnit 3.8 test framework. They are not compiled by default, compile them using the makefile in camino/test. Then execute

./runTest.sh

You can test particular packages by specifying the package name as an option, eg

./runTest.sh -numerics

Example output:

 ./runtest.sh -numerics
.........................................
.............................
Time: 7.663

OK (70 tests)

A single character is output for each test. A period means the test is OK, F means there was a failure, and E means there was an error.

Unit test organization

The unit tests live in camino/test/test. The class AllTests is what gets executed at run time. You must add your test class here in order for it to get run. The test classes themselves live in their respective packages, eg numerics, tractography, and so on.

Writing a test

Here is an example test layout:

package aPackage; 

import junit.framework.*;
import junit.extensions.*;

/**
 * Unit tests for SomeClass
 * 
 * @author <author>
 */
public class TestSomeClass extends TestCase {


    public TestSomeClass(String name) {
        super(name);
    }

    public static void main(String[] args) {
        junit.textui.TestRunner.run(suite());
    }



    public static Test suite() {
        return new TestSuite(TestSomeClass.class);
    }

    protected void setUp() {

    } 

    protected void tearDown() {

    } 

    public void testSomething() {

    }

   public void testSomethingElse() {

    }

Points to note:

Package - The package is a Camino package. At run time, classes in package X are gathered from all directories on the CLASSPATH. Thus a test class in camino/test/test/numerics is in package numerics along with the classes in camino/numerics. This means you can access protected methods within the class you're testing.
TestCase - the JUnit superclass for all tests.
Constructor - Leave this as is, use setUp() and tearDown() to initialize test resources
main - lets you run the tests in this class individually if needed.
suite() - Returns all test methods in this class. The argument is the class itself. Java will use reflection to find tests from this class. Anything that is public void and begins with the word "test" (case sensitive) will be added to the test suite.
setUp() - initialize any common resources needed by the tests. This method is optional, you can use it to initialize class variables for each test.
tearDown() - release any resources used by a test (eg clean up temp files). This is also optional, but should be included where setUp() is used.
testSomething() - here is where the tests get done. The class inherits the various assertion methods from Assert. The various assertEquals methods have a common ordering of arguments. The expected value precedes the actual value, for example: assertEquals(0.5, tensor.fa(), 1E-6);. The order is important because if the test fails, you will get a message saying "expected <0.5> but was <whatever it was>".

How the tests work

At run time, all methods in a class called public void testSomething() will be run as tests. Before a test is run, setUp() is called. After a test is run, tearDown() is called. The test method is terminated immediately if any assertion fails, however other test methods in the class will be run. If a test fails, you will get a stack trace and report after all tests are run.

Writing a good unit test

The unit testing framework offers flexibility to write interesting tests. The unit tests and the application tests should cover different domains. Application tests are regression tests - they will tell you if things are different from what they used to be, without reference to correctness. Unit tests can and should test correctness, and should be written in such a way that they are platform independent across *nix systems.

The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run datasynth and dtfit, record the results, and cut and paste them into a unit test. But this could be (and is) more easily done in ScriptTest. In the unit testing framework, we can check that things like the tensor eigenvectors and eigenvalues are within some acceptable delta of their true value. This gives us more specificity, and it allows us the flexibility to improve our solution while still being sensitive to changes that make things worse. If we improve our nonlinear optimization routine then ScriptTest is going to change, and nobody is going to wade through the text output to check whether that change in the off-diagonal elements of the diffusion tensor are correct. Unit tests to the rescue - the unit tests shouldn't fail if we do a better job of recovering the ground truth but they will fail if something is wrong.

A good test is a real effort to break the code. This is extra work than just testing trivial solutions to problems, but once written the tests will have a long life span, so it is worthwhile. A long standing bug (fixed several years ago now) existed in the FACT tractography algorithm. In the presence of multiple principal directions in a voxel, the algorithm is supposed to choose the PD most closely aligned with the tract direction. In fact, it always chose the first PD. There was a test that tracked through a synthetic fibre crossing, but it just so happened that the first PD was always the correct one to follow. The bug could have been exposed at any time by simply tracking both ways through the crossing.

Application tests

The application tests are in the file camino/test/ScriptTest. It basically runs various Camino commands and outputs results as text to stdout, or diffs against a known result. This makes it easy to write tests but makes the results machine dependent. The main purpose of these tests is for regression testing - making sure that new features don't alter existing results unexpectedly.

Because of the machine dependence of the results, it's necessary to generate results on a clean copy of the code before making local modifications. To do this, cd to camino/ and run

test/ScriptTest > ScriptTest.out.mymachine

Then make your changes and run ScriptTest again, capturing stdout to a new file as above. Then diff the output from your modified code against ScriptTest.out.mymachine.

ScriptTest format

ScriptTest is a bash script. It runs commands local to where the script is installed. It is run from the camino/ directory as test/ScriptTest.

Several variables should be used with the tests:

SCHEMEFILE=$SCRIPTDIR/bmx6.scheme
SCHEMEFILE1=$SCRIPTDIR/bmx7_ED.scheme1

D2T="double2txt 6"
F2T=float2txt
I2T=int2txt

# Random number tries to avoid conflicts
DATADIR=/tmp/caminotest_$RANDOM

Using $D2T and $F2T consistently allows you to change the number of decimal places in the output. The temporary directory DATADIR is deleted after the test completes. Many machines have /tmp on the root file system, so try not to fill it with masses of data.

Give tests a number to aid locating the output if necessary, for example:

 
# image2voxel with nii input
echo
echo "TEST 71"

image2voxel -4dimage ${SCRIPTDIR}/twoCubeRaw.nii | dtfit - ${SCHEMEFILE} | fa | $D2T

Tractography tests

Tractography tests are called from within ScriptTest. The test conditions compare expected output to that produced at run time. It is mostly robust to platform differences with a few exceptions. Currently, images are tested with diff, but this is far from an optimal solution because it tells you nothing about how different two images are. This will be fixed in a later release.

Updating ScriptTest

When you add tests to ScriptTest, you need to commit an updated ScriptTest.out.machine for your machine. This signals that the tests have been updated remotely.

Each night, the build machine at UCL runs ScriptTest. What happens next will depend on the modification time of ScriptTest.out.

If ScriptTest.out is older than any ScriptTest.out.machine, then ScriptTest.out is replaced by the build machine and the nightly download is updated.

If ScriptTest.out is newer than all ScriptTest.out.machine, then the results of ScriptTest on the build machine are checked against ScriptTest.out. If they are consistent, the nightly download update proceeds. If not, the update fails and the developers are alerted.

In other words, if you commit changes without updating ScriptTest, you should be sure that ScriptTest has not been affected by your changes (ie, your ScriptTest.out.machine is unchanged). If you commit changes with a new ScriptTest.out.machine, you are telling the build machine (and the other developers) that you have changed the behavior of ScriptTest and that your changes are correct.

Testing