fileinput requires two EOF when reading stdin

[jaraco]

BPO	15068
Nosy	@gvanrossum, @jaraco, @pitrou, @benjaminp, @bitdancer, @florentx, @vadmium, @zware, @serhiy-storchaka
Files	fileinput.patch fileinput_no_buffer.patch fileinput_no_buffer-2.7.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-03-08.16:45:33.888>
created_at = <Date 2012-06-14.15:19:34.375>
labels = ['type-bug', 'library', 'docs']
title = 'fileinput requires two EOF when reading stdin'
updated_at = <Date 2016-03-08.21:37:04.705>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2016-03-08.21:37:04.705>
actor = 'python-dev'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-03-08.16:45:33.888>
closer = 'serhiy.storchaka'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2012-06-14.15:19:34.375>
creator = 'jaraco'
dependencies = []
files = ['26018', '41225', '41226']
hgrepos = []
issue_num = 15068
keywords = ['patch']
message_count = 32.0
messages = ['162798', '162799', '162802', '162803', '162808', '162809', '162815', '162817', '162820', '162821', '162903', '162905', '162906', '162907', '162908', '162909', '162910', '162911', '162912', '162913', '162916', '162917', '162920', '255813', '256721', '256893', '256905', '260106', '261366', '261378', '261381', '261382']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'jaraco', 'pitrou', 'benjamin.peterson', 'Arfrever', 'r.david.murray', 'flox', 'docs@python', 'python-dev', 'martin.panter', 'zach.ware', 'serhiy.storchaka', 'jgeralnik']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15068'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[jaraco]

I found that fileinput.input() requires two EOF characters to stop reading input on Python 2.7.3 on Windows and Ubuntu:

PS C:\Users\jaraco> python
Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32
>>> import fileinput
>>> lines = list(fileinput.input())
foo
bar
^Z
^Z
>>> lines
['foo\n', 'bar\n']

I don't see anything in the documentation that suggests that two EOF characters would be required, and I can't think of any reason why that should be the case.

[jaraco]

I observed if I send EOF as the first character, it honors it immediately and doesn't require a second EOF.

[bitdancer]

Frankly I'm surprised it works at all, since fileinput.input() will by default read from stdin, and stdin is in turn being read by the python prompt.

I just checked 2.5 on linux, and the same situation exists there (two ^Ds are required to end the input()). I suspect we'll find the explanation in the interaction between the default behavior of fileinput.input() and the interactive prompt.

[jaraco]

FWIW, I encountered the double-EOF behavior when invoking fileinput.input from a script running non-interactively (except of course for the input() call).

[zware]

I just tested on Python 3.2, and found something interesting; it seems a ^Z character on a line that has other input read in as a character. Also, other input after an EOF on its own means you still have to do two more EOFs to end.

Python 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import fileinput
>>> lines = list(fileinput.input())
test
testing
^Z
^Z
>>> lines
['test\n', 'testing\n']
>>> lines = list(fileinput.input())
test
testing^Z
^Z
^Z
>>> lines
['test\n', 'testing\x1a\n']
>>> lines = list(fileinput.input())
testing^Z
test
^Z
testing
^Z
^Z
>>> lines
['testing\x1a\n', 'test\n', 'testing\n']

Also, the documentation for fileinput doesn't mention EOF at all.

[bitdancer]

I don't know how the EOF character works, but I wouldn't be surprised if it had to be on a line by itself to mean EOF.

If the double EOF is required when not at the interactive prompt, then there could be a long standing bug in fileinput's logic where it is doing another read after the last file is closed. Normally this wouldn't even be visible since it would just get EOF again, but when the file is an interactive STDIN, closing it doesn't really close it...

[serhiy-storchaka]

It is not only the fileinput. The same effect can be achieved by simple idiomatic code:

import sys
while True:
    chunk = sys.stdin.read(1000)
    if not chunk:
        break
    # process

[bitdancer]

That makes sense. It is a consequence of (a) buffered input and (b) the fact that EOF on stdin doesn't really close it. (And by interactive here I don't just mean Python's interactive prompt, but also the shell).

By default fileinput uses readlines with a buffer size, so it suffers from the same issue. It is only the second time that you close stdin that it gets an empty buffer, and so terminates.

Anyone want to try to come up with a doc footnote to explain this?

[serhiy-storchaka]

Note that in the rare cases, when stdio ends immediately on the limit of the read buffer, just one EOF is sufficient. In particular for read(1) one EOF is sufficient always, and for read(2) it is sufficient in about half of the cases.

[pitrou]

It is unlikely to be solvable at the Python level. Witness the raw stream's behaviour (in Python 3):

>> sys.stdin.buffer.raw.read(1000)

If you type a letter followed by ^D (Linux) or ^Z (Windows), this returns immediately:

>>> sys.stdin.buffer.raw.read(1000)
x^Db'x'

But since the result is non-empty, the buffering layer will not detect the EOF and will call read() on the raw stream again (as the 1000 bytes are not satisfied). To signal EOF to the buffered stream, you have to type ^D or ^Z *without preceding it with another character*. Try the following:

>> sys.stdin.buffer.read(1000)

You'll see that as long as you type a letter before ^D or ^Z, the read() will not return (until you type more than 1000 characters, that is):

• ^D alone: returns!
• a letter followed by ^D: doesn't return
• a letter followed by ^D followed by ^D: returns!
• a letter followed by ^D followed by a letter followed by ^D: doesn't return

This is all caused by the fact that a C read() on stdin doesn't return until either the end of line or EOF (or the requested bytes number is satisfied). Just experiment with:

>> os.read(0, 1000)

That's why I say this is not solvable at the Python level (except perhaps with bizarre ioctl hackery).

[jgeralnik]

First off, I'm a complete noob looking at the python source code for the first time so forgive me if I've done something wrong.

What if the length of the chunk is checked as well? The following code works fine:

import sys
while True:
    chunk = sys.stdin.read(1000)
    if not chunk:
        break
    # process
    if len(chunk) < 1000:
        break

Something similar could be done in the fileinput class. The patch I've attached checks if the number of bytes read from the file is less than the size of the buffer (which means that the file has ended). If so, the next time the file is to be read it skips to the next file instead.

joey@j-Laptop:~/cpython$ ./python 
Python 3.3.0a3+ (default:befd56673c80+, Jun 15 2012, 17:14:12) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fileinput
[73732 refs]
>>> lines = list(fileinput.input())
foo
bar
^D
[73774 refs]
>>> lines
['foo\n', 'bar\n']
[73780 refs]