Subversion page

Subversion and automated testing

A small story about automated testing and Subversion.

When you are developing code you are probably using a versioning system to manage the different versions of your code. With one of these systems, subversion, it is very easy to create a automated test suite around it. Currently for the development of drill we use the following setup.

Hooking into subversion

All subversion repositories have a similar layout:

README.txt  conf/  dav/  db/  format  hooks/  locks/

The hooks/ directory contains predefined hooks (scripts) which are called at different stages of a commit. The script we are interested in is post-commit.

Testing environment

We have put are testing environment inside the subversion repository, in a directory called test.

The main script is called StartTesting. This little shell script will go over each subdirectory that is named name-NN, where NN is a number and 'name' is an arbitrary name.

Each of these subdirectories contains a test script which actually performs the test. This script is called doit. Also if a file named readme is found the first line of it will be included in the status mail that is sent after the commit.

In the main test/ directory there are also two helper scripts, pre and post which perform some initialization and cleanup jobs.

How it all looks:

/test/StartTesting - the main test script
/test/pre          - initialization
/test/post         - cleanup after the tests
/test/test-01      - the first test subdir
/test/test-01/doit - the script the performs test-01
/test/test-01/readme - first line describes the test

Description of the testing process

After each commit the pre will perform a fresh checkout of the subversion repository. This newly checked out code, will then be compiled. After this step the test scripts will be run. When everything is done the post script will remove all temporary directories that we're created during testing.

Any output will be mailed back to the developers.

Status mail

After each commit we receive a mail from subversion which looks like this:

From: y 
Subject: svn 540 status

** LOG **
------------------------------------------------------------------------
r540 | x | 2004-11-24 11:50:46 +0100 (Wed, 24 Nov 2004) | 1 line

your subversion log goes here
------------------------------------------------------------------------
** ip6rev-03 SUCCES (do a reverse lookup for an ipv6 addr)

** miek-02 SUCCES (compare a query with a known correct one)

** null-04 SUCCES (do a query with for miek\0.nl)

** query-05 SUCCES (performing a bunch of queries)

** random-06 SUCCES (concatenate random data to drill)

** runtest-01 SUCCES (try to run the exe drill)

** DIFF **
We include a small diff here

So you get the test results and a log and diff mailed to you.

Conclusion

In the ideal case every bug you fix should get a test. With this setup you will only need to define the test once and all future testing is done automatically.

Repositories

NSD repository
ldns repository
autotrust repository
Drill repository