Tokenize module does not mirror "end-of-input" is newline behavior

[ammaraskar]

BPO	33899
Nosy	@terryjreedy, @gpshead, @taleinat, @benjaminp, @ned-deily, @meadori, @asottile, @ammaraskar, @miss-islington, @asmeurer
PRs	bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior #7891 [3.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-li… #8132 [2.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-li… #8133 [3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-li… #8134 [2.7] bpo-33899: Revert tokenize module adding an implicit final NEWLINE #10072 bpo-33899: add mention of this change in What's New #10073 [3.7] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) #10074 [3.6] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) #10075

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ammaraskar'
closed_at = <Date 2018-07-06.10:24:50.699>
created_at = <Date 2018-06-19.07:41:52.217>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'Tokenize module does not mirror "end-of-input" is newline behavior'
updated_at = <Date 2019-03-22.11:46:47.154>
user = 'https://github.com/ammaraskar'

bugs.python.org fields:

activity = <Date 2019-03-22.11:46:47.154>
actor = 'brechtm'
assignee = 'ammar2'
closed = True
closed_date = <Date 2018-07-06.10:24:50.699>
closer = 'taleinat'
components = ['Library (Lib)']
creation = <Date 2018-06-19.07:41:52.217>
creator = 'ammar2'
dependencies = []
files = []
hgrepos = []
issue_num = 33899
keywords = ['patch']
message_count = 27.0
messages = ['319934', '321154', '321162', '321163', '321164', '321165', '328220', '328222', '328226', '328227', '328238', '328283', '328318', '328324', '328353', '328354', '328355', '328356', '328357', '328358', '328359', '328360', '328369', '328383', '328877', '330213', '338601']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'gregory.p.smith', 'taleinat', 'benjamin.peterson', 'ned.deily', 'meador.inge', 'brechtm', 'Anthony Sottile', 'ammar2', 'miss-islington', 'asmeurer']
pr_nums = ['7891', '8132', '8133', '8134', '10072', '10073', '10074', '10075']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33899'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[ammaraskar]

As was pointed out in https://bugs.python.org/issue33766 there is an edge case in the tokenizer whereby it will implicitly treat the end of input as a newline. The tokenize module in stdlib does not mirror the C code's behavior in this case.

tokenizer.c:

~/cpython $ echo -n 'x' | ./python
----------
NAME ("x")
NEWLINE
ENDMARKER

tokenize module:

~/cpython $ echo -n 'x' | ./python -m tokenize
1,0-1,1: NAME 'x'
2,0-2,0: ENDMARKER ''

The instrumentation to have the C tokenizer dump out its tokens is mine, can provide a diff to produce that output if needed.

[taleinat]

New changeset c4ef489 by Tal Einat (Ammar Askar) in branch 'master':
bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)
c4ef489

[taleinat]

New changeset ab75d9e by Tal Einat (Ammar Askar) in branch '3.7':
[3.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8132)
ab75d9e

[taleinat]

New changeset 11c36a3 by Tal Einat (Ammar Askar) in branch '3.6':
[3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8134)
11c36a3

[taleinat]

New changeset 7829bba by Tal Einat (Ammar Askar) in branch '2.7':
[2.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (bpo-8133)
7829bba

[taleinat]

Thanks for all of your work on this, Ammar!

[asottile]

This change in behaviour is breaking pycodestyle: PyCQA/pycodestyle#786

Perhaps it shouldn't have been backported (especially all the way to python2.7?)

[taleinat]

This was backported since it was considered a bug, but you are right that it broke backwards compatibility, and perhaps shouldn't have been backported.

Still, with 3.6.6 and 3.7.1 now released, that ship has sailed.

We could perhaps revert this on the 2.7 branch, but I feel that reverting this change only on 2.7 would just cause even more confusion.

[asottile]

I'm surprised this was classified as a bug! Though that's subjective so I get that it's difficult to decide what is and what isn't ¯\(ツ)/¯