input() has trailing carriage return on windows

[kupuguy]

BPO	11272
Nosy	@kupuguy, @birkenfeld, @pitrou, @vstinner, @ezio-melotti, @bitdancer, @briancurtin
Files	issue11272.patch stdintests.py unnamed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-02-23.12:11:58.542>
created_at = <Date 2011-02-21.15:48:24.053>
labels = ['interpreter-core', 'type-bug', 'OS-windows']
title = 'input() has trailing carriage return on windows'
updated_at = <Date 2011-05-25.15:16:35.985>
user = 'https://github.com/kupuguy'

bugs.python.org fields:

activity = <Date 2011-05-25.15:16:35.985>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2011-02-23.12:11:58.542>
closer = 'vstinner'
components = ['Interpreter Core', 'Windows']
creation = <Date 2011-02-21.15:48:24.053>
creator = 'duncanb'
dependencies = []
files = ['20853', '20857', '22104']
hgrepos = []
issue_num = 11272
keywords = ['patch', '3.2regression']
message_count = 17.0
messages = ['128964', '128967', '128971', '128972', '128973', '128981', '128982', '128988', '129144', '129145', '129147', '129160', '129180', '136835', '136836', '136837', '136854']
nosy_count = 10.0
nosy_names = ['duncanb', 'georg.brandl', 'pitrou', 'vstinner', 'ezio.melotti', 'v+python', 'r.david.murray', 'brian.curtin', 'SilentGhost', 'Phoenix Fire']
pr_nums = []
priority = 'critical'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11272'
versions = ['Python 3.2']

[kupuguy]

In Python 3.2, the builtin function input() returns a string with a trailing '\r' on windows:

    C:\Python32>python
    Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print(repr(input()))
    test
    'test\r'
    >>>

This breaks code that expects the string to be stripped, e.g. 'pydoc.py -b' doesn't recognise its commands:

C:\Python32>python lib\pydoc.py -b
Server ready at http://localhost:4680/
Server commands: [b]rowser, [q]uit
server> q
Server commands: [b]rowser, [q]uit
server> b
Server commands: [b]rowser, [q]uit
server>

[ezio-melotti]

Confirmed on Python 3.2 (winxp). The problem doesn't seem to exist on 3.1.3.

[SilentGhost]

On WinXp with Python 3.2a4+ or 3.1.3 I cannot reproduce this issue.

[SilentGhost]

With py3.2 final, I can reproduce this bug with command line (as demonstrated by the OP) but not with the IDLE (for 3.2a4+ I have only command line, which I compiled myself).

[briancurtin]

bpo-10841 may be related.

[vstinner]

Confirmed on Python 3.2 (winxp).
The problem doesn't seem to exist on 3.1.3.

Can you try Python 3.1 with -u command line flag?

I changed Python 3.2 to always open all files in binary module, not only if -u flag is used. I had also to fix the parser to support \r\n newlines: it looks like I missed something in the parser.

[SilentGhost]

Can you try Python 3.1 with -u command line flag?
Yes, I can reproduce it with 3.1.3 with -u flag

[kupuguy]

Yes, it does indeed look like stdin has been opened in binary mode. Just iterating over it also gives the spurious carriage returns:

    C:\Python32>python
    Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> for line in sys.stdin:
    ...     print(repr(line))
    ...
    hello
    'hello\r\n'
    ^Z
    >>>

[vstinner]

Here is a patch to fix input() on Windows: strip also \r.

[ezio-melotti]

Is it possible to add some tests for input()?

Also the patch uses tabs instead of spaces.

[vstinner]

C:\Python32>python
    Python 3.2 ... on win32
    >>> import sys
    >>> for line in sys.stdin:
    ...     print(repr(line))
    ...
    hello
    'hello\r\n'
    ^Z

Oh yes, I confirm that there is a second bug: sys.stdin doesn't translate \r\n to \n, whereas sys.stdin is a text file.

Attached patch fixes both issues.

Is it possible to add some tests for input()?

input() has already tests, but the bug was not detected because the test uses a mockup, not a subprocess. Moreover, I don't think that it is possible to test input() for the TTY case. It is not easy to test a TTY, especially on Windows.

If anyone knows how to reproduce the two bugs with a short Python script, I can try to convert it into a test.

Also the patch uses tabs instead of spaces.

Yeah, I hate producing patches on Windows. Fixed in the new patch (prepared on Linux).

[kupuguy]

If anyone knows how to reproduce the two bugs with a short Python
script, I can try to convert it into a test.

If you don't mind kicking off some sub-processes then here's a script that shows the bugs.

I couldn't figure out how to do a script that would work on Python 3.1 but fail on Python 3.2, because I think to show the problem you have to use the shell to pipe data and Popen on Python 3.1 quotes the pipe character.