2to3 does not wrap zip correctly

[cvk]

BPO	28837
Nosy	@benjaminp, @berkerpeksag, @Mariatta, @stuarteberg
PRs	bpo-28837: Fix lib2to3 handling of map/zip/filter #24 [3.6] bpo-28837: Fix lib2to3 handling of map/zip/filter calls when fo… #2235
Files	fix-28837.patch: In lib2to3 fixers for map, zip, and filter, handle extra "trailers", e.g. zip()[0]

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-16.02:57:29.656>
created_at = <Date 2016-11-30.11:04:58.276>
labels = ['3.7', 'type-bug', 'expert-2to3']
title = '2to3 does not wrap zip correctly'
updated_at = <Date 2017-06-16.02:57:29.654>
user = 'https://bugs.python.org/cvk'

bugs.python.org fields:

activity = <Date 2017-06-16.02:57:29.654>
actor = 'Mariatta'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-16.02:57:29.656>
closer = 'Mariatta'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2016-11-30.11:04:58.276>
creator = 'cvk'
dependencies = []
files = ['46343']
hgrepos = []
issue_num = 28837
keywords = ['patch']
message_count = 8.0
messages = ['282075', '285796', '285823', '287596', '291211', '291863', '296148', '296149']
nosy_count = 5.0
nosy_names = ['benjamin.peterson', 'berker.peksag', 'Mariatta', 'cvk', 'stuarteberg']
pr_nums = ['24', '2235']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28837'
versions = ['Python 3.6', 'Python 3.7']

[cvk]

The following Python 2.7 code is not converted correctly to Python 3 by 2to3:

c = [(1, 10), (2, 20)]

# get a tuple with the first entries of every tuple in c,
# i.e. (1, 2)
x = zip(*c)[0]

print x

The result is

c = [(1, 10), (2, 20)]

# get a tuple with the first entries of every tuple in c,
# i.e. (1, 2)
x = zip(*c)[0]

print(x)

However running it with python3 gives the following exception

Traceback (most recent call last):
  File "result.py", line 7, in <module>
    x = zip(*c)[0]
TypeError: 'zip' object is not subscriptable

Tested with 2to3-2.7 and 2to3-3.4 on debian stable (jessie)

[stuarteberg]

In addition to zip(), this problem also affects map() and filter().

The problem is that the match patterns in FixZip, FixMap, and FixFilter do not allow for more than one "trailer" node. (And even if they did, their transform() methods aren't expecting it.)

For example, in the following expression, 'zip' is followed by two 'trailers', which are '(a,b)', and [0]:

zip(a,b)[0]

... but FixZip.PATTERN only expects a single trailer (the argument list), so the presence of a second trailer prevents the match: https://git.io/vMDP9

(Here's the relevant line of the grammar: https://git.io/vMDPJ)

I've written a patch that fixes this problem for zip, map, and filter, with tests. See attached.

BTW, this problem was previously reported in 21628, but that issue was incorrectly closed as a duplicate, so I'm submitting my patch here.

[stuarteberg]

Sorry for re-uploading the patch; I made some PEP-8 fixes.

[stuarteberg]

Patch submitted as github PR #24:
#24

[benjaminp]

New changeset 93b4b47 by Benjamin Peterson (Stuart Berg) in branch 'master':
bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (#24)
93b4b47

[berkerpeksag]

I think this at least needs to be backported to 3.6 (or can we close the issue now?)

[Mariatta]

New changeset 292b421 by Mariatta in branch '3.6':
bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (GH-24) (GH-2235)
292b421

[Mariatta]

Backported to 3.6.