24.7.2: Testing fails without the `TOX_PACKAGE` environment variable

[mtelka]

When I run tests without the TOX_PACKAGE environment variable set then several tests fails:

py39: commands[5]> coverage run /usr/bin/trial tests/test_examples.py
test_examples
  ExampleTests
    test_hatchling_get_version ...                                      [ERROR]
    test_noop ...                                                       [ERROR]
    test_setuppy_version ...                                            [ERROR]
    test_setuptools_bad_versionpy ...                                    [FAIL]
    test_setuptools_missing_versionpy ...                               [ERROR]
    test_setuptools_no_optin ...                                        [ERROR]
    test_setuptools_no_package ...                                      [ERROR]
    test_setuptools_version ...                                         [ERROR]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "tests/test_examples.py", line 184, in test_setuptools_bad_versionpy
    build_and_install(src)
  File "/usr/lib/python3.9/vendor-packages/twisted/trial/_synctest.py", line 355, in __exit__
    self._testCase.fail(
twisted.trial.unittest.FailTest: builtins.KeyError raised instead of BuildBackendException:
 Traceback (most recent call last):
  File "/usr/lib/python3.9/vendor-packages/twisted/trial/_asynctest.py", line 124, in _run
    d = defer.maybeDeferred(
  File "/usr/lib/python3.9/vendor-packages/twisted/internet/defer.py", line 212, in maybeDeferred
    result = f(*args, **kwargs)
  File "/usr/lib/python3.9/vendor-packages/twisted/internet/utils.py", line 223, in runWithWarningsSuppressed
    result = f(*a, **kw)
  File "tests/test_examples.py", line 184, in test_setuptools_bad_versionpy

--- <exception caught here> ---
  File "tests/test_examples.py", line 184, in test_setuptools_bad_versionpy

  File "tests/test_examples.py", line 35, in build_and_install

  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'


test_examples.ExampleTests.test_setuptools_bad_versionpy
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 190, in test_hatchling_get_version
    build_and_install(TEST_DIR.child("example_hatchling"))
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_hatchling_get_version
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 205, in test_noop
    build_and_install(TEST_DIR.child("example_noop"))
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_noop
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 54, in test_setuppy_version
    build_and_install(TEST_DIR.child("example_setuppy"))
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_setuppy_version
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 149, in test_setuptools_missing_versionpy
    build_and_install(src)
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_setuptools_missing_versionpy
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 94, in test_setuptools_no_optin
    build_and_install(src)
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_setuptools_no_optin
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 115, in test_setuptools_no_package
    build_and_install(src)
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_setuptools_no_package
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "tests/test_examples.py", line 65, in test_setuptools_version
    build_and_install(TEST_DIR.child("example_setuptools"))
  File "tests/test_examples.py", line 35, in build_and_install
    "incremental @ file://" + os.environ["TOX_PACKAGE"],
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
builtins.KeyError: 'TOX_PACKAGE'

test_examples.ExampleTests.test_setuptools_version
-------------------------------------------------------------------------------
Ran 8 tests in 0.070s

FAILED (failures=1, errors=7)

[twm]

TOX_PACKAGE points at the version of Incremental to install in the isolated build environment. Tox automatically sets it to point at the .whl, but an sdist would also work.

If you are running those tests outside of Tox you'll have to set this environment variable to point at an Incremental dist that can be installed in the venv.

Heads up: there may be some churn in this area when I address #109, so if you have to patch these tests to get them working in your environment you'll risk redoing that work soon. It might be better to skip these tests for now.

[mtelka]

Thanks for explanation. I never heard about the TOX_PACKAGE variable before. All other Python projects (over 600 as of now) I packaged for OpenIndiana works without it (although not all use tox for testing). OTOH, I assume it is legit for tests to expect the variable is set, so I'll need to deal with it for incremental somehow. For now I just skipped tests/test_examples.py tests.

Some background: I'm running tests using tox with the tox-current-env plugin installed and so tox does not build the incremental package (the wheel) for me. This means that TOX_PACKAGE is not set when I run tox to do the testing.

[twm]

Thank you, that background is very helpful! I hadn't heard of tox-current-env before. Its function is much like what I do in our CI workflow, where I build the package and then run the tests against it with tox --installpkg.

Looking at your patches, I can't quite follow why 02-tox-enable-tests.patch is necessary. Can you not use an env named test with tox-current-env?

[mtelka]

Looking at your patches, I can't quite follow why 02-tox-enable-tests.patch is necessary. Can you not use an env named test with tox-current-env?

This is not related to tox-current-env. It is there because it is usually expected that commands like tox -e py39 runs (default set of) tests against Python 3.9 and tox -e py312 against Python 3.12, etc. Since OpenIndiana is designed to support multiple Python versions installed together we need to be able to easily test using the desired and specified Python version.

Because of that I simply need the default tests available immediately and so the patch is needed.

OTOH, it should be possible to specify the Python version using something like tox -e test-py39, but our build and testing framework does not support that (yet) so I found the patch as the easiest way to solve the problem for now. Especially when the number of Python projects that needs similar patch is very low (less than 10, AFAIK).

[mtelka]

Thank you, that background is very helpful! I hadn't heard of tox-current-env before. Its function is much like what I do in our CI workflow, where I build the package and then run the tests against it with tox --installpkg.

The tox --installpkg is significantly different to tox-current-env because the former create the virtual env (installs all required packages to run tests, etc.) while the latter does not. This difference is of real value for us because testing in virtual env makes no sense for us. Your project will run in real/current environment on the operating system, so we need to test there.

[twm]

Certainly! I mean they are isomorphic: both allow running the tests against a specific Incremental package.

[glaubitz]

TOX_PACKAGE points at the version of Incremental to install in the isolated build environment. Tox automatically sets it to point at the .whl, but an sdist would also work.

I tried setting TOX_PACKAGE to point at the .whl file but the tests still fail.

[twm]

@glaubitz How exactly? (I would also expect them to fail without Internet access.)