Speedup: Add multiprocessing support to test suite

[nate-kean]

Description

Speeds up the testing suite 2-4x.¹

This pull request does two main things:

1. Refactors the test suite to allow any tests to run at the same time, and configures our CircleCI runners to run tests in parallel.
2. Makes all tests output their files to temporary directories, separate for each test.
   • Crucially, the code was changed to do this without using pushd anymore, as that was causing problems from some tests changing the Python interpreter's working directory in the middle of other tests.

Since this is off by default, documentation on enabling pytest-xdist was also added to CONTRIBUTING.

In accomplishing (1), I also had to do (2); putting every test's working files in separate directories was necessary to prevent similar tests from trying to write to the same files, which would cause them to error out and fail. Making all tests in RAiDER use temporary directories was an effort that had already been going on in RAiDER piecemeal, so now we can say that is done and dusted, too.

In the process, I ended up encountering and fixing some bugs:

• c7186f3 (#745): certain parts of RAiDER used to disobey your configured output directory, always outputting to your current working directory instead.
• 9db2a6a (#745): a piece of code was erroneously renaming any instance of "wet" to "tropo" within the configured output path. Sorry, that was my bug.
• bbbeef9 (#745): AOIs were passing the look direction in isce's format when the code was expecting it to be a string.

Motivation and Context

1. Improves developer experience by making it considerably easier/more convenient to use the test suite to test changes as you go along.
2. Improves developer experience in a different way by keeping the test suite from "polluting" their project directory with test output files. Before it could also be difficult to determine which tests were producing which files, especially since a single test could output many files across different places. Test output files are now grouped into folders labeled per test. In a typical Linux setup, they will be found in pytest-of-<username>/pytest-<run #>/<test name>.

To configure your own setup to run tests in parallel, add this to your pytest arguments:

-n <#|auto|logical>

For example, in your vscode workspace settings, you can add:

    "python.testing.pytestArgs": [
        "test",
        "-n", "logical"
    ],

This will distribute the tests across your CPU's total number of hyperthreads in vscode's pytest runner.

Note

A bug affects pytest-xdist (pytest-dev/pytest-xdist#872) that causes pytest to hang on some systems when using -n logical. If this happens to you, try an -n one less than your hyperthread count (i.e., on a system with 12 hyperthreads, set to 11).

How Has This Been Tested?

All tests continue to pass when run in parallel. I ensured that no test outputs any files to the project directory (or current working directory) anymore, through this script that watches for files being created:

#! /bin/bash
# (apt install inotify-tools)
inotifywait -r -m \
    --exclude ".*(__pycache__)|(\.git)|(pytest-cache)|(RAiDER\.egg-info).*" \
    -e create . \
    | (
        while read file_path file_event file_name; do \
            echo ${file_path}${file_name} event: ${file_event}; \
        done
    )

If you run this script and the test suite, you will find that this pull request silences all its activity.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• I have added an explanation of what your changes do and why you'd like us to include them.
• I have written new tests for your core changes, as applicable.
• I have successfully ran tests with your changes locally.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

Footnotes

1. On my machine, slow tests enabled, -n 11, run time was reduced from ~21 minutes to ~9 minutes on all of my CPU's 12 hyperthreads. On CircleCI, slow tests skipped, -n 2, "Run unit tests" time is reduced from ~17 minutes to ~4 minutes. This PR sets -n to just 2 on CircleCI to keep within the runner's RAM budget. ↩

[nate-kean]

Tests are failing on CircleCI but not on my machine... It appears that the runners are running out of memory now that they're running 32 tests at once (the number of cores they have). Assuming we don't want to pay more for a CircleCI resource class with more RAM, I can only think to find ways to reduce RAiDER's memory usage, or settle for running fewer tests in parallel.

Will try halving the worker count from 32 to 16?

[nate-kean]

Even on just 4 workers, this appears to be causing quite the amount of instability. I'm going to put this on pause and work on other things.

[nate-kean]

Docker Build failure: That looks like an out-of-memory error. Looking into it I'm noticing it's actually trying to create a RAiDER environment but is not using the conda locks. This might not actually be related to the pull request.