Let's retire pgen

[gvanrossum]

BPO	35808
Nosy	@gvanrossum, @serhiy-storchaka, @ilevkivskyi, @ammaraskar, @ethanhs, @eamanu, @pablogsal, @isidentical
PRs	bpo-35808: Retire pgen and use pgen2 to generate the parser #11814 Fix peg_generator compiler warnings under MSVC #20405

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-01.23:35:16.839>
created_at = <Date 2019-01-23.05:32:17.788>
labels = ['build', '3.8']
title = "Let's retire pgen"
updated_at = <Date 2020-05-26.04:05:22.823>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2020-05-26.04:05:22.823>
actor = 'ammar2'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-01.23:35:16.839>
closer = 'pablogsal'
components = ['Build']
creation = <Date 2019-01-23.05:32:17.788>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 35808
keywords = ['patch', 'patch', 'patch']
message_count = 5.0
messages = ['334247', '334482', '334488', '335591', '336972']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'serhiy.storchaka', 'levkivskyi', 'ammar2', 'ethan smith', 'eamanu', 'pablogsal', 'BTaskaya']
pr_nums = ['11814', '20405']
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35808'
versions = ['Python 3.8']

[gvanrossum]

Pgen is literally the oldest piece of technology in the CPython repo -- it was the first thing I wrote for Python over 29 years ago. It's not aged well, and building it requires various #if[n]def PGEN hacks in other parts of the code; it also depends more and more on CPython internals. There already is a replacement written in pure Python (Lib/lib2to3/pgen/), it just needs some glue to actually generate the graminit.[ch] files. Note that several other essential generation steps (everything listed for regen-all except regen-importlib and clinic) already depend on having a working Python interpreter around, so let's not worry about the bootstrapping process.

[eamanu]

Hello!

I can help if you consider it appropriate :-)

[gvanrossum]

@eamanu, feel free to submit a PR. I won't be available to guide you through the steps; there are other forums e.g. Zulip.

[serhiy-storchaka]

I didn't look at the PR yet, but this is a wonderful idea. I planned to do this just after rewriting generators of token-related data and code in Python (bpo-30455), but Pablo has overfoot me. Great!

[pablogsal]

New changeset 1f24a71 by Pablo Galindo in branch 'master':
bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)
1f24a71