|
TestingDevelopment.Testing HistoryHide minor edits - Show changes to markup May 03, 2011, at 07:56 PM
by
- Changed line 112 from:
The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run to:
The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run April 15, 2011, at 04:27 PM
by
- Changed line 114 from:
The best tests are a real effort to break the code. This is extra work than just testing trivial solutions to problems, but once written the tests will have a long life span, so it is worthwhile. A long standing bug (fixed several years ago now) existed in the FACT tractography algorithm. In the presence of multiple principal directions in a voxel, the algorithm is supposed to choose the PD most closely aligned with the tract direction. In fact, it always chose the first PD. There was a test that tracked through a synthetic fibre crossing, but it just so happened that the first PD was always the correct one to follow. The bug could have been exposed at any time by simply tracking both ways through the crossing. to:
A good test is a real effort to break the code. This is extra work than just testing trivial solutions to problems, but once written the tests will have a long life span, so it is worthwhile. A long standing bug (fixed several years ago now) existed in the FACT tractography algorithm. In the presence of multiple principal directions in a voxel, the algorithm is supposed to choose the PD most closely aligned with the tract direction. In fact, it always chose the first PD. There was a test that tracked through a synthetic fibre crossing, but it just so happened that the first PD was always the correct one to follow. The bug could have been exposed at any time by simply tracking both ways through the crossing. April 15, 2011, at 04:26 PM
by
- Changed line 108 from:
Writing a good testto:
Writing a good unit testApril 15, 2011, at 04:26 PM
by
- Changed line 175 from:
to:
In other words, if you commit changes without updating April 15, 2011, at 04:18 PM
by - April 15, 2011, at 04:17 PM
by
- Added lines 166-174:
When you add tests to ScriptTest, you need to commit an updated Each night, the build machine at UCL runs ScriptTest. What happens next will depend on the modification time of If If April 14, 2011, at 10:10 PM
by - April 14, 2011, at 09:14 PM
by
- Changed lines 125-168 from:
Then make your changes and run ScriptTest again, capturing stdout to a new file as above. Then diff the output from your modified code against to:
Then make your changes and run ScriptTest again, capturing stdout to a new file as above. Then diff the output from your modified code against ScriptTest formatScriptTest is a bash script. It runs commands local to where the script is installed. It is run from the Several variables should be used with the tests: SCHEMEFILE=$SCRIPTDIR/bmx6.scheme SCHEMEFILE1=$SCRIPTDIR/bmx7_ED.scheme1 D2T="double2txt 6" F2T=float2txt I2T=int2txt # Random number tries to avoid conflicts DATADIR=/tmp/caminotest_$RANDOM Using Give tests a number to aid locating the output if necessary, for example: # image2voxel with nii input echo echo "TEST 71" image2voxel -4dimage ${SCRIPTDIR}/twoCubeRaw.nii | dtfit - ${SCHEMEFILE} | fa | $D2T Tractography testsTractography tests are called from within ScriptTest. The test conditions compare expected output to that produced at run time. It is mostly robust to platform differences with a few exceptions. Currently, images are tested with Updating ScriptTestApril 14, 2011, at 08:19 PM
by
- Changed lines 100-104 from:
assertEquals(0.5, tensor.fa(), 1E-6); The order is important because if the test fails, you will get a message saying "expected <0.5> but was <whatever it was>". to:
April 14, 2011, at 08:18 PM
by
- Deleted line 103:
April 14, 2011, at 08:18 PM
by
- Changed lines 117-119 from:
The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run datasynth and dtfit, record the results, and cut and paste them into a unit test. But this could (and is) be more easily done in ScriptTest. In the unit testing framework, we can check that things like the tensor eigenvectors and eigenvalues are within some acceptable delta of their true value. This gives us more specificity, and it allows us the flexibility to improve our solution while still being sensitive to changes that make things worse. If we improve our nonlinear optimization routine then ScriptTest is going to change, and nobody is going to wade through the text output to check whether that change in the off-diagonal elements of the diffusion tensor are correct. Unit tests to the rescue - the unit tests shouldn't fail if we do a better job of recovering the ground truth but they will fail if something is wrong. to:
The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run The best tests are a real effort to break the code. This is extra work than just testing trivial solutions to problems, but once written the tests will have a long life span, so it is worthwhile. A long standing bug (fixed several years ago now) existed in the FACT tractography algorithm. In the presence of multiple principal directions in a voxel, the algorithm is supposed to choose the PD most closely aligned with the tract direction. In fact, it always chose the first PD. There was a test that tracked through a synthetic fibre crossing, but it just so happened that the first PD was always the correct one to follow. The bug could have been exposed at any time by simply tracking both ways through the crossing. April 14, 2011, at 07:40 PM
by
- Changed line 117 from:
The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run datasynth and dtfit, record the results, and cut and paste them into a unit test. But this could (and is) be more easily done in ScriptTest. In the unit testing framework, we can check that things like the tensor eigenvectors and eigenvalues are within some acceptable delta of their true value. This gives us more specificity, and it allows us the flexibility to improve our solution while still being sensitive to changes that make things worse. If we improve our nonlinear optimization routine, for example, ScriptTest is going to change, but the unit tests shouldn't fail if we do a better job of recovering the ground truth - but they will fail if something is wrong. to:
The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run datasynth and dtfit, record the results, and cut and paste them into a unit test. But this could (and is) be more easily done in ScriptTest. In the unit testing framework, we can check that things like the tensor eigenvectors and eigenvalues are within some acceptable delta of their true value. This gives us more specificity, and it allows us the flexibility to improve our solution while still being sensitive to changes that make things worse. If we improve our nonlinear optimization routine then ScriptTest is going to change, and nobody is going to wade through the text output to check whether that change in the off-diagonal elements of the diffusion tensor are correct. Unit tests to the rescue - the unit tests shouldn't fail if we do a better job of recovering the ground truth but they will fail if something is wrong. April 14, 2011, at 07:38 PM
by
- Changed lines 3-4 from:
There are two levels of testing, unit tests for individual classes and methods inside the code, and application tests that are calls to the Camino commands. All of these tests live in the to:
There are two levels of testing, unit tests for individual classes and methods inside the code, and application tests that are calls to the Camino commands. All of these tests live in the Added lines 111-118:
Writing a good testThe unit testing framework offers flexibility to write interesting tests. The unit tests and the application tests should cover different domains. Application tests are regression tests - they will tell you if things are different from what they used to be, without reference to correctness. Unit tests can and should test correctness, and should be written in such a way that they are platform independent across *nix systems. The best way to write a unit test is to compare to a ground truth solution. For example, consider the problem of estimating a diffusion tensor from some DWI data. We could just run datasynth and dtfit, record the results, and cut and paste them into a unit test. But this could (and is) be more easily done in ScriptTest. In the unit testing framework, we can check that things like the tensor eigenvectors and eigenvalues are within some acceptable delta of their true value. This gives us more specificity, and it allows us the flexibility to improve our solution while still being sensitive to changes that make things worse. If we improve our nonlinear optimization routine, for example, ScriptTest is going to change, but the unit tests shouldn't fail if we do a better job of recovering the ground truth - but they will fail if something is wrong. April 14, 2011, at 07:05 PM
by
- Changed lines 30-35 from:
The unit tests live in to:
The unit tests live in Changed lines 35-36 from:
package numerics; to:
Here is an example test layout: [@
package aPackage; Changed lines 44-49 from:
* <dl> * <dt>Purpose: Automated tests for <code>Vector3D.java</code>. * <BR><BR> * * <dt>Description: * <dd> This class is used to perform tests on <code>Vector3D</code> with JUnit 3.8. to:
* Unit tests for SomeClass Changed lines 46-53 from:
* </dl> * * @version $Id: TestVector3D.java,v 1.1 2005/09/26 10:36:49 ucacpco Exp $ * @author Philip Cook * @see numerics.Vector3D * @see <A HREF="http://www.junit.org">Junit Homepage</A> * * to:
* @author <author> Changed lines 48-51 from:
public class TestVector3D extends TestCase { public TestVector3D(String name) { to:
public class TestSomeClass extends TestCase { public TestSomeClass(String name) { Changed line 62 from:
return new TestSuite(TestVector3D.class); to:
return new TestSuite(TestSomeClass.class); Changed lines 65-109 from:
to:
protected void setUp() { } protected void tearDown() { } public void testSomething() { } public void testSomethingElse() { } @] Points to note:
assertEquals(0.5, tensor.fa(), 1E-6); The order is important because if the test fails, you will get a message saying "expected <0.5> but was <whatever it was>". How the tests workAt run time, all methods in a class called April 14, 2011, at 05:43 PM
by
- Changed line 7 from:
The unit tests are written in the JUnit test framework. They are not compiled by default, compile them using the makefile in to:
The unit tests are written in the JUnit 3.8 test framework. They are not compiled by default, compile them using the makefile in Changed lines 15-27 from:
to:
Example output: ./runtest.sh -numerics ......................................... ............................. Time: 7.663 OK (70 tests) A single character is output for each test. A period means the test is OK, F means there was a failure, and E means there was an error. Changed lines 30-31 from:
to:
The unit tests live in Added lines 38-75:
package numerics; import junit.framework.*; import junit.extensions.*; /** * <dl> * <dt>Purpose: Automated tests for <code>Vector3D.java</code>. * <BR><BR> * * <dt>Description: * <dd> This class is used to perform tests on <code>Vector3D</code> with JUnit 3.8. * * </dl> * * @version $Id: TestVector3D.java,v 1.1 2005/09/26 10:36:49 ucacpco Exp $ * @author Philip Cook * @see numerics.Vector3D * @see <A HREF="http://www.junit.org">Junit Homepage</A> * * */ public class TestVector3D extends TestCase { public TestVector3D(String name) { super(name); } public static void main(String[] args) { junit.textui.TestRunner.run(suite()); } public static Test suite() { return new TestSuite(TestVector3D.class); } April 14, 2011, at 05:26 PM
by
- Added lines 1-35:
Camino TestsThere are two levels of testing, unit tests for individual classes and methods inside the code, and application tests that are calls to the Camino commands. All of these tests live in the Unit testsThe unit tests are written in the JUnit test framework. They are not compiled by default, compile them using the makefile in ./runTest.sh
You can test particular packages by specifying the package name as an option, eg ./runTest.sh -numerics
Unit test organizationWriting a testApplication testsThe application tests are in the file Because of the machine dependence of the results, it's necessary to generate results on a clean copy of the code before making local modifications. To do this, cd to test/ScriptTest > ScriptTest.out.mymachine
Then make your changes and run ScriptTest again, capturing stdout to a new file as above. Then diff the output from your modified code against |