Skip to content

TESTING.md

Latest commit

 

History

History
276 lines (213 loc) · 9.92 KB

TESTING.md

File metadata and controls

276 lines (213 loc) · 9.92 KB

Testing of the GPSTk

Introduction

As an open source project, the source of the GPSTk is subject to intermittent updates, contributions, and corrections. The GPSTk testing process has been redesigned to build confidence in the functionality of the. Testing within the GPSTk is designed with the following distinct goals in mind:

  • Testing is repeatable with a low amount of effort.
  • Testing is distributed along with the source to support both internal testing and to assure outside users and contributors of the quality of the library.
  • Testing is designed to accommodate easy additions to the existing test suite.
  • Testing is implemented to ensure changes have not broken functionality.

All testing is performed using cmake/ctest. This allows the testing to function on all supported platforms.

The goal is to have some level of testing performed on all classes and applications in the GPSTk core. It is encouraged that tests be written for all developed code whether it be in the core or ext. In general an 'entrance criteria' for code to be included in the core is that it is tested and stable.

How to execute unit tests

  1. Using build.sh
    • $ cd ~/git/gpstk
    • $ build.sh -te
  2. Manually
    • $ cd ~/git/gpstk/build
    • $ cmake .. -DTEST_SWITCH=ON
    • $ make
    • $ ctest

How to debug the unit test results

  1. Run ctest with -V option or build.sh with the -v option
  2. Examine the test.log generated by build.sh in the build directory
  3. Examine the detailed log generated by ctest (does not require -V)
    • build/Testing/Temporary/LastTest.log
  4. Run a test report
    • $ cd ~/git/gpstk
    • $ test-report.sh

How to write application tests

  1. Write a cmake syntax script to test the program and store it in the core/tests/dir where dir corresponds to the core/apps/dir where the program source resides. Name the file after the test that is being run.
  2. Add the test to the CMakeLists.txt file in the same dir.
  3. Add any required source data to gpstk/data
  4. Add any required reference output data to gpstk/data (make the file name have something to do with the application that is being tested)

How to write class unit tests

  1. Write a C++ program in core/tests/... or ext/tests/...
    • File name starts with class name and ends with _T before the ".cpp"
  2. Add any required data to gpstk/data
  3. Modify the CMakeLists.text to build and run the tests

Current test structure

  • Unit tests for a particular GPSTk library class are often organized in a single cpp file titled by the GPSTk library class with a _T.cpp appended.
  • Unit test files are kept in gpstk/core/tests and gpstk/ext/tests
  • The individual cpp files are broken into two parts, a test class to test the GPSTk library class and a main() segment to run those tests.
  • The test class is organized into multiple public methods in which each method contains multiple assertions which test a particular feature of the GPSTk library class under test.
  • The test class might inherit from the GPSTk library class in order to access protected members for direct checking of values.
  • To facilitate reporting to the testing logs, GPSTk uses its own TestUtil class.
  • TestUtil provides standardized output containing information on the GPSTk library class being tested, feature of class being tested, test file name, line number of test in that file, pass/fail bit, and a failure message should the test have failed. It also provides the number of failures to the main() portion of the test cpp file.
  • The main() portion of the code creates the test class object and executes its methods. It then tallies the number of failures and reports it to the screen/log.
  • Data for testing is located in the gpstk/data directory. Only place data in there that is publicly releasable.
  • The file build_config.h.in is configured by the cmake process to define some functions to allow C++ programs to find this data after they are compiled.
  • The CMAKE variable GPSTK_TEST_DATA_DIR can be used to find the data from cmake
  • Applications are tested by using ctest/cmake in some clever ways. This allows the tests to not require any other scripting tools (bash/sed/diff) to perform the tests.

Examples

Application Test Example

This illustrates one test on the rmwdiff application. It requires three files to be in the gpstk/data dir. Two rinex met files; arlm200a.15m and arlm200b.15m, and an expected output file; rmwdiff1.exp.

gpstk/ext/tests/difftools/CMakeLists.txt:

...
# check differences where there is no overlap in time
add_test(NAME rmwdiff_Diff_1
         COMMAND ${CMAKE_COMMAND}
         -DTEST_PROG=$<TARGET_FILE:rmwdiff>
         -DFILE1=arlm200a.15m
         -DFILE2=arlm200b.15m
         -DTESTBASE=rmwdiff1
         -DSOURCEDIR=${GPSTK_TEST_DATA_DIR}
         -DTARGETDIR=${GPSTK_TEST_OUTPUT_DIR}
         -P ${CMAKE_CURRENT_SOURCE_DIR}/testdiff.cmake)
...

####gpstk/core/tests/difftools/testdiff.cmake:

# test that files differ

execute_process(COMMAND ${TEST_PROG} ${SOURCEDIR}/${FILE1} ${SOURCEDIR}/${FILE2}
                OUTPUT_FILE ${TARGETDIR}/${TESTBASE}.out
                RESULT_VARIABLE HAD_ERROR)
# files are expected to be different
if(!HAD_ERROR)
    message(FATAL_ERROR "Test failed")
endif()

execute_process(COMMAND ${CMAKE_COMMAND} -E compare_files
    ${TARGETDIR}/${TESTBASE}.out ${SOURCEDIR}/${TESTBASE}.exp
    RESULT_VARIABLE DIFFERENT)
if(DIFFERENT)
    message(FATAL_ERROR "Test failed - files differ")
endif()```



### Class Unit Test Example
#### gpstk/core/tests/Utilities/CMakelists.txt:

... add_executable(ValidType_T ValidType_T.cpp) target_link_libraries(ValidType_T gpstk) add_test(Utilities_ValidType ValidType_T) ...


#### gpstk/core/tests/Utilities/ValidType_T.cpp:

```c++
#include "ValidType.hpp"
#include "TestUtil.hpp"
#include <iostream>
#include <string>
#include <sstream>
#include <cmath>

class ValidType_T
{
public: 
   ValidType_T(){ eps = 1E-12;}// Default Constructor, set the precision value
   ~ValidType_T() {} // Default Desructor

   int methodTest(void)
   {
      TestUtil testFramework( "ValidType", "Various Methods", __FILE__, __LINE__ );
      std::string failMesg;

      gpstk::ValidType<float> vfloat0;

      failMesg = "Is the invalid Valid object set as valid?";
      testFramework.assert(!vfloat0.is_valid(), failMesg, __LINE__);

      failMesg = "Is the invalid Valid object's value 0?";
      testFramework.assert(std::abs(vfloat0.get_value()) < eps, failMesg, __LINE__);

      gpstk::ValidType<float> vfloat (5);

      failMesg = "Does the get_value method return the correct value?";
      testFramework.assert(vfloat.get_value() == 5, failMesg, __LINE__);

      failMesg = "Is the valid Valid object set as valid?";
      testFramework.assert(vfloat.is_valid(), failMesg, __LINE__);

      vfloat.set_valid(false);

      failMesg = "Was the valid Valid object correctly set to invalid?";
      testFramework.assert(!vfloat.is_valid(), failMesg, __LINE__);

      return testFramework.countFails();
   }

   int operatorTest(void)
   {
      TestUtil testFramework( "ValidType", "== Operator", __FILE__, __LINE__ );
      std::string failMesg;

      gpstk::ValidType<float> Compare1 (6.);
      gpstk::ValidType<float> Compare2 (6.);
      gpstk::ValidType<float> Compare3 (8.);
      gpstk::ValidType<int> Compare4 (6);
      gpstk::ValidType<float> vfloat;

      failMesg = "Are two equvalent objects equal?";
      testFramework.assert(Compare1 == Compare2, failMesg, __LINE__);

      failMesg = "Are two non-equvalent objects equal?";
      testFramework.assert(Compare1 != Compare3, failMesg, __LINE__);

      vfloat = 7.;

      testFramework.changeSourceMethod("= Operator");
      failMesg = "Did the = operator store the value correctly?";
      testFramework.assert(vfloat.get_value() == 7., failMesg, __LINE__);

      failMesg = "Did the = operator set the object as valid?";
      testFramework.assert(vfloat.is_valid(), failMesg, __LINE__);

      testFramework.changeSourceMethod("+= Operator");

      vfloat += 3.;
      failMesg = "Did the += operator store the value correctly?";
      testFramework.assert(vfloat.get_value() == 10., failMesg, __LINE__);

      failMesg = "Did the += operator change the object's valid bool?";
      testFramework.assert(vfloat.is_valid(), failMesg, __LINE__);		

      testFramework.changeSourceMethod("-= Operator");

      vfloat -= 5.;

      failMesg = "Did the -= operator store the value correctly?";
      testFramework.assert(vfloat.get_value() == 5., failMesg, __LINE__);

      failMesg = "Did the -= operator change the object's valid bool?";
      testFramework.assert(vfloat.is_valid(), failMesg, __LINE__);

      testFramework.changeSourceMethod("<< Operator");

      vfloat = 11;

      std::stringstream streamOutput;
      std::string stringOutput;
      std::string stringCompare;

      streamOutput <<  vfloat;
      stringOutput = streamOutput.str();

      stringCompare = "11";

      failMesg = "Did the << operator ouput valid object correctly?";
      testFramework.assert(stringCompare == stringOutput, failMesg, __LINE__);

      streamOutput.str("");	// Resetting stream
      vfloat.set_valid(false);

      streamOutput << vfloat;
      stringOutput = streamOutput.str();

      stringCompare = "Unknown";

      failMesg = " Did the << operator output invalid object correctly?";
      testFramework.assert(stringCompare == stringOutput, failMesg, __LINE__);

      return testFramework.countFails();
   }

private:
   double eps;
};

int main() //Main function to initialize and run all tests above
{
	int check = 0, errorCounter = 0;
	ValidType_T testClass;

	check = testClass.methodTest();
	errorCounter += check;

	check = testClass.operatorTest();
	errorCounter += check;

	std::cout << "Total Failures for " << __FILE__ << ": " << errorCounter << std::endl;

	return errorCounter; //Return the total number of errors
}