Changes between Version 3 and Version 4 of TestingGuidelines


Ignore:
Timestamp:
Feb 2, 2011, 7:08:06 PM (15 years ago)
Author:
FrederikHeber
Comment:

adapted to restructuring of regressin testsuite with commit 57beae472fa8ac008a976c744a43cdf893cd5413

Legend:

Unmodified
Added
Removed
Modified

  • TestingGuidelines

    v3 v4  
    4747Regression tests can be found in [source:tests/regression]. The basic working is: have a input file, do something with it and compare to output file against a stored one.
    4848
    49 '''Note:''' There is specific test set on ''tesselations'' in [source:tests/Tesselations] where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now. We won't go into the details of these tests here.
     49'''Note:''' There is specific test set on ''tesselations'' in [source:tests/Tesselations] where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now. The same is present for ''fragmentation'' in [source:tests/Fragmentations]. We won't go into the details of these tests here, see section ''Massive tests''.
    5050
    5151Basically, the regression test is driven via a '''testsuite''' script which is created by autotools. In [source:tests/regression/Makefile.am] you will notice it as the only element of the ''TESTSUITE'' variable given. Note however that some ''EXTRADIRS'' are specified. Hence, if you ever create a new directory for tests, add it hereto.
    5252
    53 In [source:tests/regression] you notice a lot of '''.at''' suffixed files. These are the '''a'''uto'''t'''est scripts per test category. So far, we have:
     53In [source:tests/regression] you notice a  single '''testsuite.at''', in its subfolders there are lots of '''.at''' suffixed files. These are the '''a'''uto'''t'''est scripts per test category. So far, we have:
    5454 * Analysis - analysis functons such as pair correlation, ...
     55 * Atoms - Actions that add, remove atoms, ...
    5556 * Domain - Actions that change the domain by adding empty boundary, ...
    5657 * Filling - Actions that fill the domain with molecules, such as fill void with molecule, ...
    57  * Fragmentation - fragmentation of a bonding graph into subgraphs
    58  * Graph - graph routines that recognize the bonding graph
     58 * Fragmentation - fragmentation of a bonding graph into subgraphs.
     59 * Graph - graph routines that recognize the bonding graph.
    5960 * Molecules - Actions that change molecules names, ...
    60  * Simple Configuration - tests for loading and saving files, adding, removing atoms
    61  * Specifics - very specific stuff as specifying the basis set stored in MPQC-type files
    62  * Standard Options - tests for options such as help, verbosity, version, ...
    63  * Tesselation - Actions for creating the tesselated surface of a molecule
     61 * Options - tests for options such as help, verbosity, version, ...
     62 * Parser - Actions that parse atoms contained in files into the world, also parser-specific actions.
     63 * !RandomNumbers - Action that set random number engine or distribution.
     64 * Selection - Action that (un)select atoms or molecules.
     65 * Tesselation - Actions for creating the tesselated surface of a molecule.
    6466
    65 Input and output files are stored in the subfolders of [source:tests/regression]. Each specific '''testsuite-....at''' has its own folder called alike. Have a look at '''testsuite.at''' wherefrom '''testsuite''' is constructed bu autotools. It only contains ''m4_include''s to all the test parts.
     67Input and output files are stored in the subfolders of [source:tests/regression]. Each specific '''testsuite-....at''' is contained in its own folder called alike. Have a look at '''testsuite.at''' wherefrom '''testsuite''' is constructed bu autotools. It only contains ''m4_include''s to all the test parts.
    6668
    67 These folders contain again folders with cardinal numbers, in each a '''pre''' and a '''post''' folder. '''pre''' contains input and '''post''' contains output files to test against. The numbers are arbitrary and are unique to the specific test.
     69The folder/file hierarchy is as this:
     70 * '''tests/regression''' is the folder for all regression tests.
     71 * Each test themes has its own subfolder, e.g. '''tests/regression/Analysis'''.
     72 * Therein, we need a testsuite file which m4_include's all tests to be done within that theme, called here '''testsuite-analysis.at'''.
     73 * Each test has then its own folder and should be called according to the Action under test, e.g. '''!PairCorrelation'''. Therein another autotest file called here '''testsuite-analysis-pair-correlation.at'''.
     74 * These folders contain each a '''pre''' and a '''post''' folder. '''pre''' contains input and '''post''' contains output files to test/diff against.
     75
     76The naming convention is as this:
     77 * Folder should be called as the Action NAME's, e.g. !PairCorrelation.
     78 * Files should be called as the Action TOKEN's, e.g. pair-correlation. ''However'' filenames should contain absolute path (with theme and all), e.g. ''not'' '''testsuite-pair-correlation.at''' ''but'' '''testsuite-analysis-pair-correlation.at'''.
    6879
    6980On how to write a testsuite test, we refer you to one of these '''.at''' files to get a notion and [http://www.gnu.org/software/hello/manual/autoconf/Writing-Testsuites.html#Writing-Testsuites here] to have a reference of the possible autotest commands at hand. The scheme is always the same basically:
    … …  
    8091where ''<return value>'' is some number the code returns to indicate everything worked fine. The global theme is specified only once per '''.at''' file, file ''AT_SETUP'' and ''AT_CLEANUP'' sort of embrace every specific test that you want to do.
    8192
    82 Note that testing the undo/redo-functionality of an Action is always placed into two extra tests, same small theme only with '''with Undo/Redo''' added.
    83 
    84 So, what to do step by step:
    85  1. Pick the '''testsuite-....at''' file that you think your Action fits best.
    86  1. Create your own folder within one of the subfolder list above. E.g. if 1 to 5 are used, take 6. Therein, create folders '''pre''' and '''post'''
    87  1. In '''pre''' place all the inputs files your tests needs. In '''post''' place all the outputs files as you expect your Actions to produce (note: if you create them via a different manner than by just calling your Action and before you actually create the Action, even better so. If not, ''each'' of  the stored output files should have been tested by some other means (visually at the very least).)
    88  1. Add an setup/cleanup section to the chosen '''.at''' file, place ''AT_CHECK''s inside with calls to molecuilder and your action and some fgrep for specific messages of the molecuilder run and/or diff comparing the output files.
     93Note that testing the undo/redo-functionality of an Action is always placed into the same test along with the normal functionality.
    8994
    9095Done. Test via '''make check''' which will also re-create the '''testsuite''' script.
    … …  
    126131See the [http://cppunit.sourceforge.net/doc/lastest/cppunit_cookbook.html CppUnit Cookbook] for a guide on how to write these tests.
    127132
     133==== Massive test ====
     134
     135'''Note:''' The Massive tests use purely automake constructs and has nothing to do with autotest.
     136
     137There are two more directories within '''tests''', ''tesselation'' and ''fragmentation''. There we want to test the algorithms on a whole range of different molecules to check whether all of these are working. This is too large a part for use within '''tests/regression''', hence tests as these are placed into an extra folder and have a different structure:
     138 * Have a '''Makefile.am''', where all ''TESTS'' are listed (needs to be installed in configure.ac).
     139 * Each entry in ''TESTS'' is a small file suffixed by '''.test'''.
     140 * It contains a certain call of the code along with all necessary checks, returning 0 if the test is ok, else if it failed.
     141 * In a helper file, called '''defs.in''' specific variables and functions might be implemented as helpers (needs to be installed in configure.ac)
     142
     143Automake will then dive into the subfolder and execute each file in ''TESTS'', telling whether the test was ok or failed.
     144
    128145=== How to use/call these tests? ===
    129146
    … …  
    156173The test will state '''(OK)''' when without error. Otherwise it will state '''FAIL''' and the code lines of all tests that failed
    157174
     175==== Massive test ====
     176
     177Calling the massive tests only involves stepping into the respective subfolder and calling
     178{{{
     179../../../tests/Tesselations/benzene.test
     180}}}
     181which will then execute the '''benzene.test''' test.
     182
    158183== If something is broken ==
    159184
    … …  
    167192
    168193The idea is that even when some tests are not working, '''make check''' should run through smoothly. However, it should indicate that something is amiss here, e.g.
    169  * change the message in '''AT_SETUP''' to something that contains ''BROKEN'' or alike
    170  * unit tests should
     194 * put a line just after '''AT_SETUP''' as follows:
     195{{{
     196AT_XFAIL_IF([/bin/true])
     197}}}
     198   this will tell that the test is supposed to fail and allow for the continuation of the remainder of test suite. Fix the test as soon as possible.
     199 * unit tests should print a warning message.