Repository navigation
subprocess.Popen() should allow capturing output and sending it to stdout and stderr #91378
Closed as not planned
Description
| BPO | 47222 |
|---|---|
| Nosy | @gpshead, @pprindeville |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-04-04.22:52:49.296>
labels = ['type-feature', 'library', '3.11']
title = 'subprocess.Popen() should allow capturing output and sending it to stdout and stderr'
updated_at = <Date 2022-04-05.21:08:52.185>
user = 'https://github.com/pprindeville'bugs.python.org fields:
activity = <Date 2022-04-05.21:08:52.185>
actor = 'pprindeville'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-04-04.22:52:49.296>
creator = 'pprindeville'
dependencies = []
files = []
hgrepos = []
issue_num = 47222
keywords = ['patch']
message_count = 1.0
messages = ['416703']
nosy_count = 2.0
nosy_names = ['gregory.p.smith', 'pprindeville']
pr_nums = ['32344']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue47222'
versions = ['Python 3.11']
Activity
pprindeville commentedon Apr 4, 2022
I'd like to see handlers exposes for the stdout and stderr pipe-reading threads so that I could customize what's done with the data as it's read.
I might, for instance, want to:
(1) accumulate it into a buffer;
(2) copy it onto sys.stdout or sys.stderr, respectively;
(3) send it to a logging object;
or some combination of the above, possibly all of them.
54 remaining items
bpo-47222: avoid ResourceWarnings about unclosed files (pythonGH-91378)
bpo-47222: Avoid .rstrip() where exact output is expected (pythonGH-9…
bpo-47222: Refactor stdout/stderr pipe handler into function (pythonG…
bpo-47222: Add test coverage for user-specified read_callback (python…
bpo-47222: Allow pass-thru with stdout/stderr capture (pythonGH-91378)
bpo-47222: avoid ResourceWarnings about unclosed files (pythonGH-91378)
bpo-47222: Avoid .rstrip() where exact output is expected (pythonGH-9…
bpo-47222: Refactor stdout/stderr pipe handler into function (pythonG…
bpo-47222: Add test coverage for user-specified read_callback (python…
bpo-47222: Allow pass-thru with stdout/stderr capture (pythonGH-91378)
gerion0 commentedon Oct 7, 2024
I wanted to request the same and had a hard time to find this issue (while this issue/PR is more advanced). I searched for subprocess and tee. This word is mentioned in the PR but not here. Therefore I want to leave it here to make searching easier:
"Provide a tee-mode for
subprocess.Popenwithstdoutorstderrset to something other thanNoneandsubprocess.runwithcapture_output=True"lyricaljoke commentedon Nov 6, 2024
This would be useful to have. IMO it would be best to support this generically by fully supporting using file-like objects as the
stdout,stderr, arguments. Developers could then pass in a file-like object which forwards writes to one or more locations (e.g. a string buffer and stdout). The existing interface nearly supports this by accepting certain types of file-like objects (e.g. io.StringIO, io.BytesIO etc.), but it currently only works if the file-like object actually exposes a file descriptor via thefileno()method. I have observed that this often leads to developers poorly reinventingPopen.communicate()in application code.This limitation was noted long ago in #74175, but that issue only resulted in a fix to documentation.
gpshead commentedon Nov 7, 2024
I don't doubt the utility, it is something some applications want to do (I've done it multiple times over the years as well). But I'm not convinced we want to maintain such an ability within
subprocessin the standard library itself. It adds complexity and being so abstract, would be a less than ideal implementation in many circumstances that the subprocess API caller may not be the best one to express in the context of a larger application. (async or not, threads or not, intermediate tee process or not, etc.)Design philosophy:
subprocesstoday straddles the boundary between low level and high level. Providing a mostly platform agnostic way to create and manage a child process.The reason a file descriptor from
.fileno()is required on stdin/out/err is because that is what the underlying OSes allow for connecting pipes behind the scenes so that the data flow is handled entirely by the OS without the current process being involved or even needing to exist. To support something otherwise requires us to re-invent the wheel of how to babysit pipes and copy & feed data from one place to another without blocking.Implementation wise: This ends up requiring threads for the general case. I'd rather not have the
subprocessmodule spawn threads of its own for that behind the scenes on POSIX platforms where the existence of threads can have application wide consequences (it already does spawn threads on Windows internally for.communicate()to read pipes due to a lack of event driven IO). Supporting connections to arbitrary objects supporting read or write methods would require threads regardless of if there could be more efficient ways for specific users to meet their applications particular needs.