Generate PC/python3.def by scraping headers

[zware]

BPO	23903
Nosy	@loewis, @nascheme, @pfmoore, @tjguk, @encukou, @zware, @serhiy-storchaka, @zooba
Dependencies	bpo-29058: Mark new limited C API bpo-29083: Readd PyArg_VaParse to the stable API
Files	add_python3defgen.diff 23903_with_python3.def.diff python3def.patch python3def.patch: Regenerated for review genpython3def.sh python3def-3.6.patch 23903_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-06-01.14:44:50.970>
created_at = <Date 2015-04-10.03:49:20.959>
labels = ['build', '3.8', '3.9', 'expert-C-API', 'type-feature', 'OS-windows']
title = 'Generate PC/python3.def by scraping headers'
updated_at = <Date 2021-06-01.14:44:50.969>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2021-06-01.14:44:50.969>
actor = 'petr.viktorin'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-01.14:44:50.970>
closer = 'petr.viktorin'
components = ['Build', 'Windows', 'C API']
creation = <Date 2015-04-10.03:49:20.959>
creator = 'zach.ware'
dependencies = ['29058', '29083']
files = ['38886', '38967', '45978', '46011', '46026', '46027', '46047']
hgrepos = []
issue_num = 23903
keywords = ['patch']
message_count = 36.0
messages = ['240409', '240754', '241262', '241281', '243994', '243997', '244309', '264614', '264619', '264624', '264626', '264652', '283715', '283770', '283784', '283791', '283872', '283891', '283901', '283902', '283907', '283911', '283972', '283973', '284039', '284043', '284047', '284077', '284079', '284083', '284105', '284108', '355732', '387118', '393078', '394859']
nosy_count = 9.0
nosy_names = ['loewis', 'nascheme', 'paul.moore', 'tim.golden', 'petr.viktorin', 'python-dev', 'zach.ware', 'serhiy.storchaka', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23903'
versions = ['Python 3.8', 'Python 3.9']

[zware]

The attached patch adds a script that can scrape header files to pick out PyAPI_DATA and PyAPI_FUNCs to generate PC/python3.def. It's not complete and is not integrated into the build at all yet. I'm planning to work on this further at the PyCon sprints.

Running this and comparing output to the current checked-in python3.def shows that we're not keeping it up-to-date well at all.

[zware]

Same patch, with diff to python3.def to make it available.

[loewis]

I think the approach taken in this script is incorrect. It leads to false modifications of the stable ABI, making it in fact unstable. Four examples:

PyAST_FromNode does not belong to the stable ABI, because "struct _node" doesn't belong to the stable ABI (see PEP-384 for a list of structures that belong to the stable ABI). In fact, ast.h isn't even a header file that extension modules should regularly include.

All deletions are (e.g. PyCFunction_New) are certainly incorrect: functions must never be removed from the stable ABI. In fact, Objects/methodobject.c preserves PyCFunction_New just for the sake of the stable ABI (despite there being no longer a declaration in the header file).

PyErr_SetFromWindowsErrWithFilename could belong to the stable ABI, or not, depending on the intention of stability in its addition. Since it was there since 3.0, and had not been in the stable ABI back then, I'd claim that it doesn't belong to the stable ABI (apparently, nobody has missed it). OTOH

PyExc_BrokenPipeError might be a useful addition to the stable ABI. However, it was not there in 3.0, so the header files ought to version-guard its declaration, but currently don't.

So I prefer if the def file keeps being maintained manually; I don't mind if there is a tool that checks for inconsistencies, leaving their resolution to a maintainer.

[zooba]

That would be the preferred way to do it, but we got no traction from anyone at PyCon or on python-dev when we tried to get people to look at what's changed.

I suspect we haven't marketed the stable API well enough to have a lot of users, and the build tools still mark everything as 3.x-specific rather than 3-specific. Keeping it as a bare minimum interface has its value, but it's not unreasonable to expect that the functions available from the headers under Py_LIMITED_API are actually exported by python3.dll.

[zware]

What about including this in Tools/scripts/patchcheck.py? I think that might strike a good enough balance between making changes noticeable and not automatically changing the def file.

As far as all the differences this has uncovered, I'll try to make a patch to #ifdef them in/out.

[zooba]

Could the latter break people already using the stable ABI on other platforms? Does it even work for others? I don't know what the equivalent of python3.dll would be.

If it doesn't exist outside of Windows, then assuming that everything not in python3.def isn't stable is fine. Otherwise I think we need to go the other way.

[zooba]

It might be nice to have a test that tries to import all of the stable ABI at build time so that the buildbots break if someone adds a new API and doesn't think about what they've done.

As an aside, it'd be nice for the script to take the version numbers as args so we can integrate it into the MSBuild steps (which already know the version) and avoid having to update this each time we update the Python version.

[zware]

Unassigning since I haven't made it anywhere with this in a year; if anybody else would like to pick it up, feel free. Otherwise I'll get to it some day :)

[serhiy-storchaka]

This tool would be very useful, since new API often accidentally marked as stable API in header files. The script can play just deliberative role, warn if new unstable API is not wrapped with "#ifndef Py_LIMITED_API" or if stable API is not documented. May be we need separate list of public API in more regular format for comparison with header, documentation, python3.def, refcounts.dat, etc.

There is a code for parsing preprocessor instructions in Argument Clinic. May be it can be reused here.

[zooba]

As I recall, the tool is ready to go. The problem is the argument over what to do with all the APIs that currently are incorrect. Zach had a list at PyCon last year, and the consensus seemed to be that "someone" should figure out whether each API should be stable or not. Since Zach understandably doesn't want to be that someone, there's really no way to reliably tool this process when there are a number of existing violations.

(FWIW, I believe we should just add them to the stable list, as that's the only way to guarantee source compatibility. They've already shipped in 3.5 and people can use them on non-Windows OSs.)