bpo-35975: Support parsing earlier minor versions of Python 3 #12086

gvanrossum · 2019-02-28T00:44:51Z

This adds a feature_version flag to ast.parse() (documented) and compile() (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if feature_version is 5 or 6, the hacks for the async and await keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than NAME tokens that the parser driver recognizes.)

https://bugs.python.org/issue35975

gvanrossum · 2019-02-28T04:08:08Z

I could use some help finding the cause of the one remaining test failure -- test_parser.py computes the size of tree nodes differently somehow. [UPDATE: Fixed it, I think.]

gvanrossum · 2019-02-28T23:56:28Z

The main thing left to do is adding tests. There are some tests in typed_ast that I can migrate over. But first I think we should complete the debate over on bpo about whether this is desirable at all.

ilevkivskyi

Thanks! This looks very clean, I just have few minor suggestions.

(I also there is a merge conflict now.)

ilevkivskyi · 2019-03-02T23:54:43Z

Lib/ast.py

-    return compile(source, filename, mode, flags)
+    return compile(source, filename, mode, flags,
+                   dont_inherit=False,
+                   optimize=-1,


Why we now need to pass these two extra arguments? Maybe add a comment?

Hm, those are the default values that I somehow copied when cleaning this up. I'll remove them -- we just need feature_version.

ilevkivskyi · 2019-03-03T00:02:48Z

Python/ast.c

+        /* Async comprehensions only allowed in Python 3.6 and greater */
+        if (is_async && c->c_feature_version < 6) {
+            ast_error(c, n,
+                    "Async comprehensions are only supported in Python 3.6 and greater");


Indentation here and in several other places is a bit unusual, I would rather justify it after (.

OK, fixed. My guess is that at some point in the past that function had had a name that was 2 characters shorter. :-)

ilevkivskyi · 2019-03-03T00:03:49Z

Python/ast.c


+    if (is_async && c->c_feature_version < 5) {
+        ast_error(c, n,
+                  "Async functions are only supported in Python 3.5 and greater");


Here indentation looks good.

Visible behavior: - Issue error for `X @ Y` (matrix multiply) if c_feature_version < 5 - Add optional feature_version kw arg to ast.parse() (default -1 which implies PY_MINOR_VERSION) - Add feature_version: int = -1 to compile() (via Argument Clinic); this sets cf_feature_version to the given value if >= 0, else defaults to PY_MINOR_VERSION Implementation: - Add PyAST_obj2mod_ex(): like PyAST_obj2mod() but with feature_version arg; the latter calls the former with PY_MINOR_VERSION - Add cf_feature_version to PyCompilerFlags structure; initialized to PY_MINOR_VERSION everywhere - Add c_feature_version to struct compiling; initialize from cf_feature_version - Add 'c' argument to get_operator() - In builtin eval() and exec(), default to PY_MINOR_VERSION TODO: - Put version-dependent ASYNC/AWAIT keyword scanning back - Reject async functions, await expressions, and async for/with in minor versions < 5 - Reject async comprehensions in minor versions < 6 - Reject underscores in numeric literals in minor versions < 6 - Reject variable annotations in minor versions < 6 - Reject `X @= Y` in minor versions < 5

This is everything currently in typeshed except await expressions (but it does reject async functions etc.): - Reject async functions and async for/with in minor versions < 5 - Reject async comprehensions in minor versions < 6 - Reject underscores in numeric literals in minor versions < 6 - Reject variable annotations in minor versions < 6 - Reject `X @= Y` in minor versions < 5

This adds: - Add ASYNC/AWAIT tokens back to Grammar and regenerate - Recognize async/await keywords conditionally if feature_version < 7 - Reject await expressions if feature_version < 5 - Docs for ASYNC/AWAIT tokens and for ast.parse(..., feature_version=N)

The PyST_Object header in parsermodule.c became one int larger because it contains a PyCompilerFlags struct, which grew extra space for the st_feature_version field. Took me long enough!

… not regular keywords

gvanrossum · 2019-03-05T19:38:27Z

OK, this is ready for final review and merge. (Sorry that the rebase lost some of the review history, I'm used to different tooling.)

gvanrossum · 2019-03-05T19:50:12Z

(Well, I promised tests. Upcoming. Docs are already done.)

ilevkivskyi

Thanks for the updates! All looks good, I have few optional suggestions.

ilevkivskyi · 2019-03-07T11:49:06Z

Doc/library/ast.rst


+   Also, setting ``feature_version`` to the minor version of an
+   earlier Python 3 version will attempt to parse using that version's
+   grammar.  For example, setting ``feature_version=4`` will allow


Maybe add that 4 is the lowest supported value?

ilevkivskyi · 2019-03-07T14:02:59Z

Lib/test/test_type_comments.py

+        return ast.parse(source, type_comments=True,
+                         feature_version=feature_version)
+
+    def parses(self, source, minver=lowest, maxver=highest, expected_regex=""):


parses is not very descriptive and is easy to confuse with parse, maybe parse_all, or parse_all_versions?

ilevkivskyi · 2019-03-07T14:04:01Z

Lib/test/test_type_comments.py

+fstring = """\
+a = 42
+f"{a}"
+"""


Maybe add underscores in numeric literals for completeness?

gvanrossum added the DO-NOT-MERGE label Feb 28, 2019

the-knights-who-say-ni added the CLA signed label Feb 28, 2019

bedevere-bot added the awaiting merge label Feb 28, 2019

ilevkivskyi approved these changes Mar 3, 2019

View reviewed changes

gvanrossum added 10 commits March 5, 2019 11:36

Add blurb

1851eaf

Check flags isn't NULL before dereferencing (this broke regen-importlib)

00fdd08

'async' and 'await' are no longer plain keywords -- fix test_keyword.py

543ab75

Fix test_parser.py

9f4cb23

The PyST_Object header in parsermodule.c became one int larger because it contains a PyCompilerFlags struct, which grew extra space for the st_feature_version field. Took me long enough!

Manually add async and await to kwlist in keywords.py, since they are…

855e07b

… not regular keywords

Simplify compile() call in ast.parse()

ee95fef

Fix indentation of many ast_error() calls

ae36b7b

gvanrossum force-pushed the feature-version branch from 87c80c1 to ae36b7b Compare March 5, 2019 19:36

gvanrossum changed the title ~~bpo-35975: [WIP] Support parsing earlier minor versions of Python 3~~ bpo-35975: Support parsing earlier minor versions of Python 3 Mar 5, 2019

gvanrossum removed the DO-NOT-MERGE label Mar 5, 2019

Add tests for feature_version

bae211c

ilevkivskyi approved these changes Mar 7, 2019

View reviewed changes

gvanrossum added 3 commits March 7, 2019 12:06

Clarify range of feature_version in docs.

dbb08cc

Rename parses() to parse_all()

c8c84f4

Add test for underscore in number

e925175

gvanrossum added the 🤖 automerge label Mar 7, 2019

miss-islington merged commit 495da29 into master Mar 7, 2019

bedevere-bot removed the awaiting merge label Mar 7, 2019

zware deleted the feature-version branch April 10, 2019 02:26

Uh oh!

bpo-35975: Support parsing earlier minor versions of Python 3 #12086

bpo-35975: Support parsing earlier minor versions of Python 3 #12086

Uh oh!

Conversation

gvanrossum commented Feb 28, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented Feb 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented Feb 28, 2019

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gvanrossum commented Mar 5, 2019

Uh oh!

gvanrossum commented Mar 5, 2019

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gvanrossum commented Feb 28, 2019 •

edited by bedevere-bot

Loading

gvanrossum commented Feb 28, 2019 •

edited

Loading