`CliRunner(mix_stderr=False)` does not capture all writes to `stderr`

[RhetTbull]

When CliRunner(mix_stderr=False) is used, the output of print("foo", file=sys.stderr) is not captured. click.echo("foo", err=True) is captured (and this is tested in the test_stderr() test) but the output from print() is not captured.

Modifying tests/testing.py::test_stderr as follows demonstrates the bug:

def test_stderr():
    @click.command()
    def cli_stderr():
        click.echo("stdout")
        click.echo("stderr", err=True)
        print("sys.stderr", file=sys.stderr)

    runner = CliRunner(mix_stderr=False)

    result = runner.invoke(cli_stderr)

    assert result.output == "stdout\n"
    assert result.stdout == "stdout\n"
    assert result.stderr == "stderr\nsys.stderr\n"

    runner_mix = CliRunner(mix_stderr=True)
    result_mix = runner_mix.invoke(cli_stderr)

    assert result_mix.output == "stdout\nstderr\nsys.stderr\n"
    assert result_mix.stdout == "stdout\nstderr\nsys.stderr\n"

    with pytest.raises(ValueError):
        result_mix.stderr

    @click.command()
    def cli_empty_stderr():
        click.echo("stdout")

    runner = CliRunner(mix_stderr=False)

    result = runner.invoke(cli_empty_stderr)

    assert result.output == "stdout\n"
    assert result.stdout == "stdout\n"
    assert result.stderr == ""

Running this test, produces:

__________________________________________________________________________________ test_stderr ___________________________________________________________________________________

    def test_stderr():
        @click.command()
        def cli_stderr():
            click.echo("stdout")
            click.echo("stderr", err=True)
            print("sys.stderr", file=sys.stderr)

        runner = CliRunner(mix_stderr=False)

        result = runner.invoke(cli_stderr)

        assert result.output == "stdout\n"
        assert result.stdout == "stdout\n"
>       assert result.stderr == "stderr\nsys.stderr\n"
E       AssertionError: assert 'stderr\n' == 'stderr\nsys.stderr\n'
E           stderr
E         - sys.stderr

tests/test_testing.py:316: AssertionError
============================================================================ short test summary info =============================================================================
FAILED tests/test_testing.py::test_stderr - AssertionError: assert 'stderr\n' == 'stderr\nsys.stderr\n'
=============================================================================== 1 failed in 0.09s ================================================================================

The fix appears to be flusing stderr in src/click/testing.py:

--- a/src/click/testing.py
+++ b/src/click/testing.py
@@ -437,6 +437,7 @@ class CliRunner:
                 if self.mix_stderr:
                     stderr = None
                 else:
+                    sys.stderr.flush()
                     stderr = outstreams[1].getvalue()  # type: ignore

With this fix, and the modified test_stderr(), all tests pass.

Environment:

• OS: macOS Ventura 13.5.1, Intel x86
• Python version: Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct 2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
• Click version: version = "8.2.0.dev"

If you confirm this is a bug and not the intended behavior I am happy to submit a PR.

[davidism]

This isn't a bug. When using print in general, you have to flush to be sure all output is sent to stderr. That's one of the reasons Click provides echo instead.

[RhetTbull]

That's true with regular files but according to the python docs sys.stderr is always line buffered (unless used with -u flag) which means that CliRunner() is changing the expected behavior of print(*args, file=sys.stderr) (for Python >= 3.9)

When interactive, the stdout stream is line-buffered. Otherwise, it is block-buffered like regular text files. The stderr 
stream is line-buffered in both cases. You can make both streams unbuffered by passing the [-u]
(https://docs.python.org/3/using/cmdline.html#cmdoption-u) command-line option or setting the 
[PYTHONUNBUFFERED](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED) environment 
variable.

Changed in version 3.9: Non-interactive stderr is now line-buffered instead of fully buffered.

The following code demonstrates that sys.stderr is line buffered:

import sys
import time

def main():
    """Demonstrate that stderr is line buffered.
    This will pause 3 seconds, then print "Hello World!\n" twice to stderr then pause for 3 seconds and print "Done.".
    """
    # Hello World! will be printed after 3 seconds (stderr is line buffered) so "Hello " is not printed until the new line in the next print statement causes the buffer to be flushed
    print("Hello ", end="", file=sys.stderr)
    time.sleep(3)
    print("World!", file=sys.stderr)

    # Hello World! will be printed immediately
    print("Hello World!", file=sys.stderr)
    time.sleep(3)
    print("Done.")

if __name__ == "__main__":
    main()

[kdeldycke]

This has been fixed in v8.2.1 by #2933.