subprocess seems to use local encoding and give no choice

[mark-summerfield]

BPO	6135
Nosy	@gpshead, @amauryfa, @ncoghlan, @pitrou, @vstinner, @mark-summerfield, @merwok, @bitdancer, @mightyiam, @andyclegg, @cjerdonek, @vadmium, @eryksun, @zooba, @davispuh
PRs	bpo-6135: Fix subprocess.check_output() doc to mention changes in 3.6 #5564 [3.7] bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) #5572 [3.6] bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) #5573
Files	subprocess.patch: Add encoding and errors to subprocess.Popen. Based against trunk. subprocess3.patch: Add encoding and errors to subprocess.Popen. Based against py3k. test_subprocess3.py.patch: unittests for encoding parameter subProcessTest.py: Workaround example. Calls itself as a sub-process. 6135_1.patch 6135_2.patch 6135_3.patch 6135_4.patch 6135_5.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2016-09-07.03:18:30.672>
created_at = <Date 2009-05-28.07:08:42.275>
labels = ['type-feature', 'library']
title = 'subprocess seems to use local encoding and give no choice'
updated_at = <Date 2018-02-07.02:11:33.939>
user = 'https://github.com/mark-summerfield'

bugs.python.org fields:

activity = <Date 2018-02-07.02:11:33.939>
actor = 'gregory.p.smith'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-09-07.03:18:30.672>
closer = 'steve.dower'
components = ['Library (Lib)']
creation = <Date 2009-05-28.07:08:42.275>
creator = 'mark'
dependencies = []
files = ['14293', '14294', '14295', '28760', '44386', '44389', '44398', '44404', '44408']
hgrepos = []
issue_num = 6135
keywords = ['patch']
message_count = 66.0
messages = ['88466', '89094', '89293', '89322', '89325', '89332', '89333', '97090', '97092', '97093', '111181', '111466', '123020', '123024', '148025', '148066', '148484', '148494', '148495', '148496', '148498', '168191', '168213', '180157', '265785', '265793', '265812', '265822', '274467', '274468', '274469', '274481', '274489', '274492', '274497', '274509', '274510', '274517', '274518', '274519', '274561', '274562', '274566', '274574', '274642', '274644', '274646', '274648', '274649', '274651', '274653', '274665', '274699', '274736', '274737', '282291', '282294', '282298', '304049', '304060', '304118', '304119', '304126', '311756', '311757', '311760']
nosy_count = 20.0
nosy_names = ['gregory.p.smith', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'mark', 'eric.araujo', 'segfaulthunter', 'Arfrever', 'r.david.murray', 'srid', 'mightyiam', 'andrewclegg', 'chris.jerdonek', 'python-dev', 'martin.panter', 'eryksun', 'steve.dower', 'berwin22', 'davispuh']
pr_nums = ['5564', '5572', '5573']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6135'
versions = ['Python 3.6']

[mark-summerfield]

When I start a process with subprocess.Popen() and pipe the stdin and
stdout, it always seems to use the local 8-bit encoding.

I tried setting process.stdin.encoding = "utf8" and the same for stdout
(where process is the subprocess object), but to no avail.

I also tried using shell=True since on Mac, Terminal.app is fine with
Unicode, but that didn't work.

So basically, I have programs that output Unicode and run fine on the
Mac terminal, but that cannot be executed by subprocess because
subprocess uses the mac_roman encoding instead of Unicode.

I wish it were possible to specify the stdin and stdout encoding that is
used; then I could use the same one on all platforms. (But perhaps it is
possible, and I just haven't figured out how?)

[srid]

Related discussion thread: https://answers.launchpad.net/bzr/+question/63601

[amauryfa]

I propose to add two parameters (encoding, error) to the
subprocess.Popen function.

• python 2.x could build and return codecs.StreamReader objects
• python 3.x would just pass these parameters to io.TextIOWrapper

I'll try to come with a patch.

[segfaulthunter]

I wrote a patch to add encoding and error to subprocess.Popen in Python
2.7 (trunk).

[segfaulthunter]

Cosmetic update.

[pitrou]

Two things:

1. The argument should be called errors for consistency with open()
   and TextIOWrapper(), not error
2. You should add some unit tests.

[segfaulthunter]

Should we also cover the unusual case where stdout, stderr and stdin
have different encodings, because now we are assuming the are all the same.

[mark-summerfield]

I agree with Florian Mayer that the encoding handling should be
stream-specific. You could easily be reading the stdout of some third
party program that uses, say, latin1, but want to do your own output in,
say, utf-8.

One solution that builds on what Amaury Forgeot d'Arc has done (i.e.,
the encoding and errors parameters) by allowing those parameters to
accept either a single string (as now), or a dict with keys 'stdin',
'stdout', 'stderr'. Of course it is possible that the client might not
specify all the dict's keys in which case those would use the normal
default (local 8-bit etc.)

[amauryfa]

I don't understand. How is the subprocess stdout related to the main
program output?
Stream-specific encoding could be useful for subprocesses that expect
latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
support this.

[mark-summerfield]

On Thu, Dec 31, 2009 at 1:30 PM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

I don't understand. How is the subprocess stdout related to the main
program output?
Stream-specific encoding could be useful for subprocesses that expect
latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
support this.

Yes, you're right.

(What I had in mind was a scenario where you read one process's stdout
and wrote to another process's stdin; but of course using your errors
& encoding arguments this will work because there'll be two separate
process objects each of which can have its encoding and errors set
separately.)

[BreamoreBoy]

Ran new unit test before and after patching subprocess on Windows Vista against 3.1 debug maintenance release, all ok apart from this at end of latter.

File "test\test_subprocess.py", line 568, in test_encoded_stderr
self.assertEqual(p.stderr.read(), send)
AssertionError: 'ï[32943 refs]\r\n' != 'ï'

I'm sure I've seen a ref to this somewhere, can anyone remember where?

[BreamoreBoy]

In 2.7 and py3k test_subprocess has a class BaseTestCase which has a assertStderrEqual method. These don't exist in 3.1. I believe that the py3k code will need to be backported to 3.1. Can this be done on this issue, or do we need a new one to keep things clean?

[vstinner]

About the topic:

subprocess seems to use local 8-bit encoding and gives no choice
I don't understand that: by default, Python 2 and Python 3 use byte strings, so there is no encoding (nor error handler).

I don't see how you can get unicode from a process only using subprocess. But with Python 3, you can get unicode if you set universal_newlines option to True.

So for Python 2, it's a new feature (get unicode), and for Python 3, it's a new option to specify the encoding. The title should be changed to something like "subprocess: add an option to specify stdin, stdout and/or stderr encoding and errors" and the type should be changed to "feature request".

Or am I completly wrong?