Python 3.5 can't compile AST involving PEP 448 unpacking

[dabeaz]

BPO	24975
Nosy	@brettcannon, @terryjreedy, @ncoghlan, @larryhastings, @benjaminp, @serhiy-storchaka, @1st1
Files	issue24975.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-09-02.19:51:32.261>
created_at = <Date 2015-09-01.01:13:16.515>
labels = ['library', 'type-crash', 'release-blocker']
title = "Python 3.5 can't compile AST involving PEP 448 unpacking"
updated_at = <Date 2015-09-04.21:46:54.617>
user = 'https://bugs.python.org/dabeaz'

bugs.python.org fields:

activity = <Date 2015-09-04.21:46:54.617>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2015-09-02.19:51:32.261>
closer = 'yselivanov'
components = ['Library (Lib)']
creation = <Date 2015-09-01.01:13:16.515>
creator = 'dabeaz'
dependencies = []
files = ['40311']
hgrepos = []
issue_num = 24975
keywords = ['patch']
message_count = 11.0
messages = ['249442', '249448', '249451', '249452', '249498', '249515', '249517', '249563', '249564', '249565', '249838']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'terry.reedy', 'ncoghlan', 'larry', 'benjamin.peterson', 'dabeaz', 'python-dev', 'serhiy.storchaka', 'yselivanov']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue24975'
versions = ['Python 3.5', 'Python 3.6']

[dabeaz]

The compile() function is not able to compile an AST created from code that uses some of the new unpacking generalizations in PEP-448. Example:

code = '''                                                                     
a = { 'x':1, 'y':2 }
b = { **a, 'z': 3 }
'''

# Works
ccode = compile(code, '', 'exec')

# Crashes
import ast
tree = ast.parse(code)
ccode = compile(tree, '', 'exec')

# ----------------------------------

Error Traceback:

Traceback (most recent call last):
  File "bug.py", line 11, in <module>
    ccode = compile(tree, '', 'exec')
ValueError: None disallowed in expression list

Note: This bug makes it impossible to try generalized unpacking examples interactively in IPython.

[1st1]

Good find, David. Thanks!

Please review the attached patch.

[larryhastings]

Good catch! Please file a pull request via bitbucket.

[ncoghlan]

Patch looks good to me. I was initially going to query the changes to the break/continue AST tests, but then realised they were needed to avoid a compile error for break/continue outside a loop now that we ensure the resulting AST can be used to generate code.

[brettcannon]

Do we not have a test that compiles the entire stdlib using the ast module? We used to have one for the old compiler package. If don't have such a test we should probably add it even if it requires a -u option.

[1st1]

Larry,

I created a PR for you: https://bitbucket.org/larry/cpython350/pull-requests/10/issue-24975-fix-ast-compilation-for-pep/diff

Brett,

Should we do that in a separate issue targeting 3.6 only?

[brettcannon]

Created issue bpo-24981 to track the test case idea.

[larryhastings]

Merged. Please forward-merge to 3.5.1 and 3.6, thanks!

[python-dev]

New changeset e9df8543d7bc by Yury Selivanov in branch '3.5':
Issue bpo-24975: Fix AST compilation for PEP-448 syntax.
https://hg.python.org/cpython/rev/e9df8543d7bc

New changeset ea79be5b201e by Yury Selivanov in branch '3.5':
Merge 3.5 heads (issue bpo-24975)
https://hg.python.org/cpython/rev/ea79be5b201e

New changeset 1dcd7f257ed8 by Yury Selivanov in branch 'default':
Merge 3.5 (issue bpo-24975)
https://hg.python.org/cpython/rev/1dcd7f257ed8

[1st1]

Thanks Nick and Larry for code reviews and merge!

[terryjreedy]

Upon writing a proof-of-concept file for bpo-24981, I discovered an stdlib file that gave the same error.

C:/programs/Python35/Lib\distutils/_msvccompiler.py
ValueError: None disallowed in expression list

I presume the applied fix will apply to this also.