Cannot test output to `stderr`

[dwt]

I am not entierely sure this is a bug or more a consequence of how the testing support is integrated into click.

Here is my scenario: I am writing a small shell script that outputs an error message to stderr by calling sys.exit(some_string). This is fine, if perhaps a bit unintuitive that I have to check result.exception for this instead of result.stderr.

However earlier the program configures the logging package with logging.basicConfig(level=logging.INFO). In my testing, as well as how I read the documentation this should mean that all output from logging.warning('Fnord') and similar logging should be printed to stderr.

They are however not accessible via result.stderr.

Several reasons come to mind: I am testing via pytest, which does log capturing, so I guess that it just captures the log messages, due to the code in question being called in process, not via subprocess.run().

import logging

import click
from click.testing import CliRunner


def test_reproduction():
    @click.command
    def main():
        logging.basicConfig(level=logging.INFO)
        logging.error('fnord')
    runner = CliRunner(mix_stderr=False)
    result = runner.invoke(main, [])
    assert 'fnord' in result.stderr

I'm not quite sure how to achieve this, but I would like to not have to care in unit tests about what pytest or another test framework does and get the log messages the way I configured them. Because in essence this means that it is not easily testable that logging is wired correctly to print to stderr in the cli application.

Environment:

• Python version: 3.11.2
• Click version: 8.1.3
• pytest version: 7.2.2

[davidism]

If you run the given example outside Pytest, it passes. I'm guessing this is some weird thing with pytest's output capture, there are various other issues about pytest and pdb.

[davidism]

Additionally, if you replace logging with sys.stderror.write, print, click.echo, etc., the test passes. So this is likely something that pytest is doing with log captures.

[dwt]

Yes, this is most likely about pytest capturing the warnings.

I guess the only real fix would be to actually run the invoked process in another python process. Not something I gather you are willing to do?

[kdeldycke]

This is going to be addressed in v8.4.0 by #3391 which introduce a new capture="fd" mode to CliRunner.