Generate all tokens related code and docs from Grammar/Tokens

[serhiy-storchaka]

BPO	30455
Nosy	@vstinner, @benjaminp, @bitdancer, @meadori, @serhiy-storchaka, @matrixise, @albertjan
PRs	bpo-30455: Generate tokens related C code and docs from token.py. #1860 Add additional generated files to .gitattributes #9343 bpo-30455: Generate all token related code and docs from Grammar/Tokens. #10370 bpo-35224: PEP 572 Implementation #10497

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2018-12-22.09:26:57.225>
created_at = <Date 2017-05-24.12:21:49.802>
labels = ['interpreter-core', '3.8', 'type-feature', 'library']
title = 'Generate all tokens related code and docs from Grammar/Tokens'
updated_at = <Date 2018-12-24.14:48:06.911>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-12-24.14:48:06.911>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2018-12-22.09:26:57.225>
closer = 'serhiy.storchaka'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2017-05-24.12:21:49.802>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 30455
keywords = ['patch']
message_count = 12.0
messages = ['294350', '294356', '294361', '294363', '294753', '294754', '294833', '329375', '330053', '332195', '332205', '332459']
nosy_count = 7.0
nosy_names = ['vstinner', 'benjamin.peterson', 'r.david.murray', 'meador.inge', 'serhiy.storchaka', 'matrixise', 'Albert-Jan Nijburg']
pr_nums = ['1860', '9343', '10370', '10497']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30455'
versions = ['Python 3.8']

[serhiy-storchaka]

Currently Lib/token.py is generated from Include/token.h. This contradicts common practice when the C code is generated from the Python code (see for example opcode.py and sre_constants.py). In additional the table in Parser/tokenizer.c should be manually supported matching Include/token.h.

Generating Include/token.h and Parser/tokenizer.c from Lib/token.py would be simpler and more reliable.

[vstinner]

I like the idea.

[matrixise]

I can work on it

[serhiy-storchaka]

I already write a patch.

[serhiy-storchaka]

PR 1860 makes following files be generated from token.py:

• Include/token.h
• Parser/token.c. New file containing the array of token names _PyParser_TokenNames, and functions PyToken_OneChar(), PyToken_TwoChars(), PyToken_ThreeChars(), moved from Parser/tokenizer.c.
• Doc/library/token-list.inc. New file containing the list of token.py constants, it is included in Doc/library/token.rst.

New Makefile target regen-token regenerates these files.

The dict EXACT_TOKEN_TYPES that maps operator strings to token names now is automatically generated and moved from tokenize.py to token.py. Tokens COMMENT, NL and ENCODING used only in tokenize.py now are added in token.py as in bpo-25324.

[albertjan]

I think this covers all the changes from PR bpo-1608. Looks a lot nicer too, building it every time from the make file.

You may want to add to the docs that token.py is now the source of the tokens.

[serhiy-storchaka]

The regular expression tokenize.Funny also can be generated. Information is not enough for distinguish between Operator, Bracket and Special, but seems this isn't needed.

Some token names can be generated from Grammar/Grammar. But needed an additional mapping for relations between token strings and names ('+' <-> PLUS, etc).

[serhiy-storchaka]

Alternate PR 10370 generates all files from a single file Grammar/Tokens using a single script Tools/scripts/generate_token.py.

In addition, the script doesn't write files when the content is not changed. Thus it can be used with read-only sources.