python --help: -u is misdocumented as binary mode

[arigo]

BPO	28647
Nosy	@ncoghlan, @vstinner, @gareth-rees, @berkerpeksag, @serhiy-storchaka, @eryksun
PRs	bpo-28647: Update -u documentation #1655 [3.6] bpo-28647: Update -u documentation #1796 [3.5] bpo-28647: Update -u documentation #1797 [3.6] bpo-28647: Update -u documentation #3954 bpo-28647: Update -u documentation after bpo-30404 #3961
Files	issue28647.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-13.12:17:18.132>
created_at = <Date 2016-11-09.10:26:05.573>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'python --help: -u is misdocumented as binary mode'
updated_at = <Date 2017-10-13.13:23:37.089>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2017-10-13.13:23:37.089>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-13.12:17:18.132>
closer = 'berker.peksag'
components = ['Interpreter Core']
creation = <Date 2016-11-09.10:26:05.573>
creator = 'arigo'
dependencies = []
files = ['45463']
hgrepos = []
issue_num = 28647
keywords = ['patch']
message_count = 32.0
messages = ['280390', '280391', '280392', '280393', '280394', '280405', '280665', '280666', '289649', '293961', '304136', '304139', '304140', '304141', '304147', '304206', '304207', '304211', '304216', '304228', '304231', '304237', '304240', '304242', '304251', '304255', '304256', '304259', '304331', '304333', '304334', '304336']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'vstinner', 'gdr@garethrees.org', 'berker.peksag', 'serhiy.storchaka', 'eryksun']
pr_nums = ['1655', '1796', '1797', '3954', '3961']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28647'
versions = ['Python 3.6', 'Python 3.7']

[arigo]

python3.5 --help gives this information:

-u : unbuffered binary stdout and stderr, stdin always buffered

However, stdout and stderr are actually always opened in text mode, and print() always expects a string and never a bytes object. This usage of "binary" in the --help is in contradiction with the usage of "binary" in the description of files (e.g. help(open)).

[vstinner]

Right. Moreover, "unbuffered" is wrong. It's line buffered: sys.stdout.buffer is directly a io.FileIO object, but TextIOWrapper only calls buffer.write() when the message contains a newline character. It's not (currently) possible to have a fully unbuffered stdout.

[eryksun]

It's not (currently) possible to have a fully unbuffered stdout.

Why doesn't create_stdio also pass write_through = Py_True when Py_UnbufferedStdioFlag is set? This would immediately pass writes through to the FileIO object, even without containing a newline (i.e. it sets text_needflush in _io_TextIOWrapper_write_impl).

[vstinner]

I don't recall, I would have to search in old issues for the rationale. But
the explanation is probably performance, reduce the number of syscalls.

[serhiy-storchaka]

The write_through argument was added only in 3.3. I think this is a good idea to pass write_through = Py_True when Py_UnbufferedStdioFlag is set. Using -u means that performance is not important (that is why this is not default behavior).

[vstinner]

Would it make sense to have two modes: line buferred and unbuffered,
as the C function setvbuf() for stdout?

[gareth-rees]

The output of "python3.5 --help" says:

-u : unbuffered binary stdout and stderr, stdin always buffered;
     also PYTHONUNBUFFERED=x
     see man page for details on internal buffering relating to '-u'

If you look at the man page as instructed then you'll see a clearer
explanation:

-u   Force  the  binary  I/O  layers  of  stdout  and  stderr  to  be
     unbuffered.  stdin is always buffered.  The text I/O layer  will
     still be line-buffered.

For example, if you try this:

python3.5 -uc 'import sys,time;w=sys.stdout.buffer.write;w(b"a");time.sleep(1);w(b"b");'

then you'll see that the binary output is indeed unbuffered as
documented.

The output of --help is trying to abbreviate this explanation, but I
think it's abbreviated too much. The explanation from the man page
seems clear to me, and is only a little longer, so I suggest changing
the --help output to match the man page.

[gareth-rees]

Here's a patch that copies the text for the -u option from the man page to the --help output.

[ncoghlan]

I just ran into this discrepancy working on the test cases for PEP-538 - +1 for Gareth's suggested approach of just aligning the --help output with the man page.

[serhiy-storchaka]

bpo-30404 makes stdout and stderr truly unbuffered when run with -u.

[berkerpeksag]

Pull request for bpo-30404 has been merged so we only need the documentation patch for the 3.6 branch (unfortunately 3.5 is now in security-fix-only mode) I've opened PR 3954.