Testing code

TODO: update to refer to more recent discussions on testing in python

• use numpy.testing functions for numerical assessment

How should I test my code?

Tests are an opportunity to invent the interface(s) you want. Write the test for a method before you write the method: often, this helps you figure out what you would want to call it and what parameters it should take. It's OK to write the tests a few methods at a time, and to change them as your ideas about the interface change. However, you shouldn't change them once you've told other people what the interface is.

Never treat prototypes as production code

It's fine to write prototype code without tests to try things out, but when you've figured out the algorithm and interfaces you must rewrite it with tests to consider it finished. Often, this helps you decide what interfaces and functionality you actually need and what you can get rid of.

"Code a little test a little"

For production code, write a couple of tests, then a couple more tests, then a couple more methods, change some of the names or generalise some of the functionality. If you have a huge amount of code where 'all you have to do is write the tests', you're closer to 30% done than 90%. Testing vastly reduces the time spent debugging, since whatever went wrong has to be in the code you wrote since the your last run of the test suite. And remember to use Python's interactive interpreter for quick checks of syntax and ideas.

Run the test suite when you change anything. Even if a change seems trivial, it will only take a couple of seconds to run the tests and then you'll be sure. This can eliminate long and frustrating debugging sessions where the change turned out to have been made long ago, but didn't seem significant at the time.

Some pytest pointers

• We put tests in a separate file for each module. Name the test file test_module_name.py. Keeping the tests separate from the code reduces the temptation to change the tests when the code doesn't work, and makes it easy to verify that a completely new implementation presents the same interface (behaves the same) as the old.

• Use established pytest fixtures. These are pytest magic which provides common capabilities defined in one place (cogent3 has tests/conftest.py). Look at those before writing your own.

• Test all the methods and functions in your code. We strive for 100% test coverage! You should assume that any function or method that is untested has bugs. The convention for naming tests is test_method_name. Any leading and trailing underscores on the method name can be ignored for the purposes of the test; however, all tests must start with the literal substring test for pytest to find them. If the method is particularly complex, or has several discretely different cases you need to check, use test_method_name_suffix, e.g. test_init_empty, test_init_single, test_init_wrong_type, etc. for testing __init__.

• Write good comments and /or docstrings for all your test functions.

  Good docstrings:

    NumberList.var should raise ValueError on empty or 1-item list
    NumberList.var should match values from R if list has >2 items
    NumberList.__init__ should raise error on values that fail float()
    FrequencyDistribution.var should match corresponding NumberList var
  

  Bad docstrings:

    var should calculate variance           # lacks class name, not descriptive
    Check initialization of a NumberList    # doesn't say what's expected
    Tests of the NumberList initialization. # ditto
  

• It is much more important to test several small cases that you can check by hand than a single large case that requires a calculator. Don't trust spreadsheets for numerical calculations! Find a reliable source.

• Make sure you test all the edge cases: what happens when the input is None, or '', or 0, or negative? What happens at values that cause a conditional to go one way or the other? Does incorrect input raise the right exceptions? Can your code accept subclasses or superclasses of the types it expects? What happens with very large input?

• *Use the pytest.mark.paramtrization decorator to generate combinations of conditions for the same test code. This is a powerful way of expanding the number of conditions that are checked with minimal additional code for you. These eliminate the need for looping within your tests!