fileinput should use stdin.buffer for "rb" mode

[brandon-rhodes]

BPO	21075
Nosy	@vstinner, @serhiy-storchaka, @MojoVampire
Files	21075-fileinput-stdin.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2014-05-14.18:17:41.145>
created_at = <Date 2014-03-27.13:42:43.555>
labels = ['type-bug', 'library']
title = 'fileinput should use stdin.buffer for "rb" mode'
updated_at = <Date 2016-01-02.20:45:09.623>
user = 'https://bugs.python.org/brandon-rhodes'

bugs.python.org fields:

activity = <Date 2016-01-02.20:45:09.623>
actor = 'python-dev'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2014-05-14.18:17:41.145>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2014-03-27.13:42:43.555>
creator = 'brandon-rhodes'
dependencies = []
files = ['34880']
hgrepos = []
issue_num = 21075
keywords = ['patch']
message_count = 6.0
messages = ['214952', '215001', '216331', '218558', '218559', '257362']
nosy_count = 6.0
nosy_names = ['vstinner', 'brandon-rhodes', 'python-dev', 'serhiy.storchaka', 'josh.r', 'sam.kimbrel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21075'
versions = ['Python 3.3', 'Python 3.4']

[brandon-rhodes]

In Python 3, fileinput.input() returns str lines whether the data is
coming from stdin or from a list of files on the command line. But if
input(mode='rb') is specified, then its behavior becomes inconsistent:
lines from stdin are delivered as already-decoded strings, but data
from files is delivered (correctly) as bytes.

The solution may be that, if a "b" is anywhere in the mode, then input()
should read from the bytes stdin.buffer data source instead of from
stdin.

Otherwise the "rb" mode is rather useless since you can wind
up getting text from it anyway depending on how you are invoked.

[MojoVampire]

There is a similar, (unfixed?) bug, bpo-14156, in argparse as well. Seems like a common failing in the move to Python 3; std*.buffer was introduced, but none of the places that use it were updated, so they all became str only.

[samkimbrel]

Patch attached that checks for 'b' in self._mode and sets self._file to sys.stdin.buffer as appropriate.

[python-dev]

New changeset 7e640fefc9c1 by Serhiy Storchaka in branch '3.4':
Issue bpo-21075: fileinput.FileInput now reads bytes from standard stream if
http://hg.python.org/cpython/rev/7e640fefc9c1

New changeset 4041d4077a85 by Serhiy Storchaka in branch 'default':
Issue bpo-21075: fileinput.FileInput now reads bytes from standard stream if
http://hg.python.org/cpython/rev/4041d4077a85

[serhiy-storchaka]

Thank you Sam for your contribution.

[python-dev]

New changeset ded1336bff49 by R David Murray in branch '3.5':
bpo-22709: Use stdin as-is if it does not have a buffer attribute.
https://hg.python.org/cpython/rev/ded1336bff49