Semi-Automatic testing of Starlab:    ( a proposal )
----------------------------------

Date:    27-feb-2001
Authors: Peter Teuben
	 Simon Portegies Zwart

We propose an automated testing procedure, modeled after the
"Testfile" approach NEMO has been using. This procedure places a small
makefile-type file, called Testfile, in various directories in which
testing should take place. This has the advantage - under assumptions
and limitations how you write these Testfiles - that testing can be
done locally, but is easily (see below) done for the whole system
and scales trivially as the system grows. This unlike a single
shell script that tests and compares various executables that are
in the system.

It uses an object-oriented approach in the sense that the Testfile
must be able to tell the caller what it can test, what output it
produces etc.

A general top level script will then hunt for Testfile's, and then
automatically test (and build if need be) the whole system.

We have implemented this example in $STARLAB_PATH/src/nody/dyn/init

A script 'testsuite' in $STARLAB_PATH/sbin can excersize the Testfiles.
It will create and write output in a fresh $STARLAB_PATH/tmp/testXXXXX
directory (XXXXX is the PID of the testsuite process)

For examples:

    testsuite -h	reminder to all the options

    testsuite -n		dryrun, just shows where Testfiles live
    testsuite			run all tests, shows OK or FAILED for each
    testsuite -d		run all tests, but also diff/cmp output files



You can also change some global variables, e.g. NBODY or SEED, if you want to
test the whole system under a different stress level. Or use it for benchmarking
if you consider running the whole suite for say NBODY=2048. The testsuite

-------------------------------------------------------------------------------    
    
Authors of Testfiles are encouraged to test their work. For their
ease, they can make an alias

      alias test 'make -f Testfile'

and do
    
      test all			

to run the tests, and 

      test clean

to clean up your mess.

--------------------------------------------------------------------------------

In summary, 

   testsuite

itself will run through a set of executables, and will complain if any of them
did not seem to run fine (but does not know if data differ from a saved baseline).
One can still difference the produced "testsuite.log" file with a baseline
version to see how much things have changed.

On a second level, and this is certainly ok within a CPU family, you can run

   testsuite -d

and compare the output on a more detailed level. It is now also easy to test
programs individually.


--------------------------------------------------------------------------------

Here are some caveats:

  - differencing data (diff or cmp) has the danger that they will not exactly
    match on different cpus (e.g. sparc vs. intel) due to differences in
    floating point rounding
  - random number generators (although used with fixed seed) can differ on
    different cpus (16bit vs. 32bit vs. 64bit)
  - there is no procedure yet (flag to testsuite) to maintain testdata.
    (we propose to put a set in
    

--------------------------------------------------------------------------------
Here is an example Testfile: (from src/node/dyn/init/Testfile)


# BEGIN Testfile

# --- standard local variables

DIR  = src/node/dyn/init
TEST = _mkplummer _mkmass
NEED = mkmass mkplummer
OUT  = mkplummer.test mkmass.test

# temporary variables, can be overridded

NBODY = 1024
SEED = 42
BASEDIR = .

# --- probably don't modify this, maybe clean needs to do more

help:
	@echo $(DIR)

test:
	@echo $(TEST)

need:
	@echo $(NEED)

clean:
	@echo Cleaning $(DIR)
	@rm -f $(OUT)

out:
	@echo $(OUT)

all:	$(TEST)

diff:
	-@for i in $(OUT); do\
	(diff $$i $(BASEDIR)); done

	

# ---- write your individual tests below, in usual makefile language

_mkplummer: mkplummer.test

mkplummer.test:
	mkplummer -n $(NBODY) -s $(SEED) > mkplummer.test

_mkmass:	mkmass.test

mkmass.test:
	mkmass -l 1 -u 100 -s $(SEED) < mkplummer.test  > mkmass.test

# END Testfile