tokenize.generate_tokens doesn't always return logical line

[duncf]

BPO	5028
Nosy	@jackdied, @miss-islington
PRs	bpo-5028: fix doc bug for tokenize #11683 bpo-5028: fix doc bug for tokenize #11683 bpo-5028: fix doc bug for tokenize #11683 bpo-5028: fix doc bug for tokenize #11683 bpo-5028: Fix up rest of documentation for tokenize documenting line #13686

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-30.19:33:23.206>
created_at = <Date 2009-01-22.01:39:01.871>
labels = ['easy', 'type-bug', '3.8', 'docs']
title = "tokenize.generate_tokens doesn't always return logical line"
updated_at = <Date 2019-05-30.22:06:44.692>
user = 'https://bugs.python.org/duncf'

bugs.python.org fields:

activity = <Date 2019-05-30.22:06:44.692>
actor = 'miss-islington'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-05-30.19:33:23.206>
closer = 'cheryl.sabella'
components = ['Documentation']
creation = <Date 2009-01-22.01:39:01.871>
creator = 'duncf'
dependencies = []
files = []
hgrepos = []
issue_num = 5028
keywords = ['patch', 'patch', 'patch', 'patch', 'easy']
message_count = 4.0
messages = ['80353', '84222', '343995', '344008']
nosy_count = 5.0
nosy_names = ['jackdied', 'LambertDW', 'duncf', 'docs@python', 'miss-islington']
pr_nums = ['11683', '11683', '11683', '11683', '13686']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5028'
versions = ['Python 3.8']

[duncf]

According to the documentation for tokenize.generate_tokens:

"The generator produces 5-tuples with these members: the token type; the
token string; a 2-tuple (srow, scol) of ints specifying the row and
column where the token begins in the source; a 2-tuple (erow, ecol) of
ints specifying the row and column where the token ends in the source;
and the line on which the token was found. The line passed (the last
tuple item) is the logical line; continuation lines are included."

It seems though that the "logical line" -- the last element of the tuple
is the physical line unless the token being returned spans beyond the
end of the line. As an example, consider a test file test.py:

foo = """
%s """ % 'bar'

>>> import pprint, tokenize
>>> pprint.pprint(list(tokenize.generate_tokens(open('test.py').readline)))
[(1, 'foo', (1, 0), (1, 3), 'foo = """\n'),
 (51, '=', (1, 4), (1, 5), 'foo = """\n'),
 (3, '"""\n%s """', (1, 6), (2, 6), 'foo = """\n%s """ % \'bar\'\n'),
 (51, '%', (2, 7), (2, 8), '%s """ % \'bar\'\n'),
 (3, "'bar'", (2, 9), (2, 14), '%s """ % \'bar\'\n'),
 (4, '\n', (2, 14), (2, 15), '%s """ % \'bar\'\n'),
 (0, '', (3, 0), (3, 0), '')]
>>> 

Since there is only one logical line, I would expect the first 6 tokens
to have the same 5th element.

[jackdied]

+1 for a docbug. The last item is always the physical line and not the
logical line. Some other examples:

if True and \
   False: pass

if (True and
    False): pass

[miss-islington]

New changeset 1e36f75 by Miss Islington (bot) (Andrew Carr) in branch 'master':
bpo-5028: fix doc bug for tokenize (GH-11683)
1e36f75

[miss-islington]

New changeset 2a58b06 by Miss Islington (bot) (Anthony Sottile) in branch 'master':
bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)
2a58b06