Put back the ability to parse files where async/await aren't keywords

[gvanrossum]

BPO	35975
Nosy	@gvanrossum, @scoder, @encukou, @serhiy-storchaka, @1st1, @ilevkivskyi, @ethanhs, @miss-islington, @mayankasthana, @msullivan
PRs	bpo-35975: Support parsing earlier minor versions of Python 3 #12086 bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags #21021 [3.9] bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags #21022 [3.8] bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags #21023

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2020-06-29.16:40:15.196>
created_at = <Date 2019-02-12.00:23:38.530>
labels = ['interpreter-core', 'type-bug', '3.8']
title = "Put back the ability to parse files where async/await aren't keywords"
updated_at = <Date 2020-06-29.16:40:15.195>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2020-06-29.16:40:15.195>
actor = 'gvanrossum'
assignee = 'gvanrossum'
closed = True
closed_date = <Date 2020-06-29.16:40:15.196>
closer = 'gvanrossum'
components = ['Interpreter Core']
creation = <Date 2019-02-12.00:23:38.530>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 35975
keywords = ['patch']
message_count = 26.0
messages = ['335277', '336785', '336786', '336788', '336789', '336790', '336795', '336800', '336831', '336879', '337243', '337432', '343749', '343786', '371938', '371943', '371949', '371974', '371975', '371983', '372006', '372019', '372484', '372485', '372486', '372602']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'scoder', 'petr.viktorin', 'serhiy.storchaka', 'yselivanov', 'levkivskyi', 'ethan smith', 'miss-islington', 'masthana', 'msullivan']
pr_nums = ['12086', '21021', '21022', '21023']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35975'
versions = ['Python 3.8']

[gvanrossum]

Now that the ast module can be used to parse type comments, there’s one more feature I’d like to lobby for – a flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4).

This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether async and await are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful.

The implementation in typed_ast takes the form of a feature_version flag which is set to the minor Python version to be parsed. Setting it to 4 or lower would make async/await never keywords, setting it to 5 or 6 would make them conditional per PEP-492, and setting it to 7 or higher would make these unconditional keywords. A few minor uses of the same flag also reject f-strings, annotated assignments (PEP-526), and matrix multiply for versions where those don't exist.

But I don't need all of those -- I really just need to be able to select the 3.5/3.6 rules for conditional async/await keywords, since all the other features are purely backwards compatible, whereas with async/await there is legal (and plentiful!) code that uses these as variable or function names that should be supported in 3.5/3.6 mode.

Of course having it be a (minor) version number would still be more future-proof -- if say in 3.10 we add a match expression using some kind of conditional keyword hack, we might have to dust it off.

Note that much of what I'm asking for would effectively roll back https://bugs.python.org/issue30406 -- sorry Jelle! (Though there's more to it -- Serhiy's introduction of Grammar/Token followed, and I would still need to thread some kind of flag all the way from ast.parse() to tokenizer.c.

[1st1]

Well, it's not just rolling back async/await from being keywords. Since 3.7 it's possible to create async generator expressions in non-async functions. This wasn't possible to do with old hacks on the lexer. So if you want to revert the change you risk losing some functionality we enabled in 3.7

[gvanrossum]

That should still work. The strategy is as follows:

• For Python 3.7 or higher, 'async' and 'await' are *always* recognized.

• For Python 3.5 and 3.6, the hacks from the PEP are used.

[1st1]

Actually, I think we could revert to old lexer hacks in 3.8 if we modify them to treat 'async for' tocken sequence as one meta-tocken.

[gvanrossum]

But why would we? I already have a working solution. (Literally reverting Jelle's PR won't work anyway, because Serhiy changed how regen-token works.)

[1st1]

But why would we? I already have a working solution.

I've heard some complaints that it's hard to migrate to Python 3.7 because async and await are keywords (although I think by now all popular libraries have migrated already), so in case you ever considered to revert that decision I think we could actually make it work. But for the current PR you're working on we don't need that, you're right.

[gvanrossum]

And the same tokenizer trick that detects 'async def' could detect 'async for' easily. See https://github.com/python/cpython/pull/12086/files#diff-30b8266a4285de981f8b1b82a8cc6231R1418

[1st1]

And the same tokenizer trick that detects 'async def' could detect 'async for' easily.

Exactly. I just never thought that we could support async generator expressions with the kind of hacks in tokenizer we had in 3.5-3.6. FWIW I still believe that it's better for async & await tokens to be proper keywords, but in case we ever realize we want them to become "soft" keywords we apparently have options.

[serhiy-storchaka]

Would not be simpler to just drop the support of Python versions <3.7 in new MyPy versions?

[gvanrossum]

Would not be simpler to just drop the support of Python versions <3.7 in new MyPy versions?

Not really -- mypy has a lot of users who run it over (very) large code bases that can't easily be upgraded. Basically mypy has to support all Python versions that haven't reached their end of life yet. (And it's also important that mypy not be constrained to the same Python version as the target code.)

I suppose we could just stick with the existing typed_ast that supports 3.4 through 3.7, but we actually have a use case for the end_lineno and end_col_offset fields that were just added to ast in 3.8, and in general we'd like to support any future grammar elements and ast features.

[gvanrossum]

Everyone watching, the PR is now ready for review!