| 1 | /*
|
|---|
| 2 | * Project: MoleCuilder
|
|---|
| 3 | * Description: creates and alters molecular systems
|
|---|
| 4 | * Copyright (C) 2010 University of Bonn. All rights reserved.
|
|---|
| 5 | * Please see the LICENSE file or "Copyright notice" in builder.cpp for details.
|
|---|
| 6 | */
|
|---|
| 7 |
|
|---|
| 8 | /**
|
|---|
| 9 | * \file regression-tests.dox
|
|---|
| 10 | *
|
|---|
| 11 | * This file contains explanation how to write, launch and use regression tests.
|
|---|
| 12 | *
|
|---|
| 13 | * Created on: Oct 11, 2011
|
|---|
| 14 | * Author: heber
|
|---|
| 15 | */
|
|---|
| 16 |
|
|---|
| 17 |
|
|---|
| 18 | /**
|
|---|
| 19 | * \page regressiontest "Regression tests"
|
|---|
| 20 | *
|
|---|
| 21 | * Regression test with this project are used to check on the functionality of
|
|---|
| 22 | * all Actions. The launch the MoleCuilder from the command line with a given
|
|---|
| 23 | * action and option and basically diff the output against a stored file with
|
|---|
| 24 | * the desired result. This should be the behavior of more than 90% of all
|
|---|
| 25 | * regression tests.
|
|---|
| 26 | *
|
|---|
| 27 | * \section Directory structure
|
|---|
| 28 | *
|
|---|
| 29 | * They are contained in the source folder \b tests/regression. There are
|
|---|
| 30 | * categories containing a distinct folder for each action. The folder for the
|
|---|
| 31 | * test of a specific Action has the following structure:
|
|---|
| 32 | * \li a folder \b pre contains all files required as input to the test.
|
|---|
| 33 | * \li a folder \b post contains all output files against which we compare all
|
|---|
| 34 | * or a number of resulting files.
|
|---|
| 35 | * \li a test script \b testsuite-....at which is included in a similarly-
|
|---|
| 36 | * named test script one directory level higher.
|
|---|
| 37 | *
|
|---|
| 38 | * \section Adding a new test
|
|---|
| 39 | *
|
|---|
| 40 | * Adding a new regression tests consists of the following items:
|
|---|
| 41 | * -# Create a new folder in a matching category
|
|---|
| 42 | * -# Create therein subfolders \b pre and \b post
|
|---|
| 43 | * -# Create a test script where the name begins with \b testsuite- and contains
|
|---|
| 44 | * category and the action token. See other test scripts to get an idea on how
|
|---|
| 45 | * to write these and also look here (http://www.gnu.org/s/hello/manual/autoconf/Using-Autotest.html#Using-Autotest).
|
|---|
| 46 | * -# In the command \a AT_KEYWORD which must be present you should given the
|
|---|
| 47 | * token of the Action as it would be called from the command line (see below for
|
|---|
| 48 | * the reason).
|
|---|
| 49 | * -# Include your testscript in the one present one directory-level up.
|
|---|
| 50 | * -# Also include your script in \b tests/scripts/Makefile.am such that the
|
|---|
| 51 | * testsuite is automatically re-compiled when one of the test files has changed.
|
|---|
| 52 | *
|
|---|
| 53 | * \section Launching all tests
|
|---|
| 54 | *
|
|---|
| 55 | * Launching all regression tests is as simple as:
|
|---|
| 56 | * -# Enter the build directory
|
|---|
| 57 | * -# There, enter \b tests/regression
|
|---|
| 58 | * -# Run
|
|---|
| 59 | * \code make check \endcode
|
|---|
| 60 | * (at you liberty with option \a -j8 or similar for running the tests in parallel.
|
|---|
| 61 | *
|
|---|
| 62 | * \section Launching a specific tests
|
|---|
| 63 | *
|
|---|
| 64 | * Launching a single or just some of the tests is only a little bit more complicated.
|
|---|
| 65 | * There are two options: either by the test number which however changes when new
|
|---|
| 66 | * tests are added, and by keywords.
|
|---|
| 67 | *
|
|---|
| 68 | * Then proceed as follows:
|
|---|
| 69 | * -# Enter the build directory
|
|---|
| 70 | * -# There, enter \b tests/regression
|
|---|
| 71 | * -# Run
|
|---|
| 72 | * \code ../../../tests/regression/testsuite <option> AUTOTEST_PATH="<buildpath>/src" \endcode,
|
|---|
| 73 | * where \a <option> is explained in the subsections below and \a <buildpath> is the build
|
|---|
| 74 | * path (i.e. the variable \a AUTOTEST_PATH should contain the path to the executable).
|
|---|
| 75 | *
|
|---|
| 76 | * \subsection ... by number
|
|---|
| 77 | *
|
|---|
| 78 | * Tests can be launched by specifying their test number, e.g. then \a <option>
|
|---|
| 79 | * might be \a 283 or \a 283-284 or \a 283,285-286 or alike
|
|---|
| 80 | *
|
|---|
| 81 | * \subsection .. by keyword
|
|---|
| 82 | *
|
|---|
| 83 | * Tests may as well be launched by some keywords, e.g. each Action has a specific
|
|---|
| 84 | * token which is also one of its keyword (see above for the policy). I.e. we may
|
|---|
| 85 | * launch the test on fill-void via the \a <option> \a -k \a fill-void. Also
|
|---|
| 86 | * multiple keywords may be given.
|
|---|
| 87 | *
|
|---|
| 88 | * \section Inspecting test results
|
|---|
| 89 | *
|
|---|
| 90 | * The testsuite can be launched with the additional option of \a -d which leaves the
|
|---|
| 91 | * directory of the test present even though the test has passed for inspection.
|
|---|
| 92 | *
|
|---|
| 93 | * If a test fails, in whatever way it was launched, will leave in the build directory
|
|---|
| 94 | * a folder \b tests/regression/testsuite.dir/<nr> where \a <nr> is the number of the
|
|---|
| 95 | * test (padded maybe with some zeros).
|
|---|
| 96 | *
|
|---|
| 97 | */
|
|---|
