ABI breakage between Python 3.7.4 and 3.7.5: change in PyGC_Head structure

[jd]

BPO	39599
Nosy	@vstinner, @ned-deily, @methane, @jd

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-02-29.22:49:21.442>
created_at = <Date 2020-02-10.12:54:16.771>
labels = ['expert-C-API', '3.7', 'invalid']
title = 'ABI breakage between Python 3.7.4 and 3.7.5: change in PyGC_Head structure'
updated_at = <Date 2020-04-09.15:20:16.710>
user = 'https://github.com/jd'

bugs.python.org fields:

activity = <Date 2020-04-09.15:20:16.710>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-29.22:49:21.442>
closer = 'vstinner'
components = ['C API']
creation = <Date 2020-02-10.12:54:16.771>
creator = 'jd'
dependencies = []
files = []
hgrepos = []
issue_num = 39599
keywords = []
message_count = 17.0
messages = ['361689', '361695', '361696', '361811', '361812', '361813', '361814', '361866', '361867', '361868', '363029', '363031', '363033', '363252', '363253', '363255', '366065']
nosy_count = 4.0
nosy_names = ['vstinner', 'ned.deily', 'methane', 'jd']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39599'
versions = ['Python 3.7']

[jd]

As I've reported originally on the python-dev list, there seems to be an ABI breakage between 3.7.4 and 3.7.5.

https://mail.python.org/archives/list/python-dev@python.org/message/J2FGZPS5PS7473TONJTPAVSNXRGV3TFL/

The culprit commit is 8766cb7

This happens on a custom C module (built via Cython) when using including <internal/pystate.h> with -DPy_BUILD_CORE. I'm not sure it'd happen otherwise.

I've tried to provide a minimal use case, but since it seems to be a memory overflow, the backtrace does not make any sense and it's hard to reproduce without the orignal code.

[vstinner]

The latest 3.7 release is Python 3.7.6: Dec. 18, 2019. So at least two 3.7 release are impacted. Moreover, Python 3.8.0 and 3.8.1 have also been released with the change.

[vstinner]

You can find the rationale for this change in two issues:

• bpo-27987
• bpo-36618

First, it was a warning in clang ubsan (Undefined Behavior Sanitizer).
Then ctypes started to crash when Python was compiled with clang. It
means that compiling Python 3.7 with clang also had the issue.

The quick fix was to compile Python with -fmax-type-align=8 when clang
was detected.

But it was a signal that something was wrong in Python on x86-64:
Python didn't respect the x86-64 ABI.

[vstinner]

The PyObject structure is part of the stable ABI, but PyGC_Head is not.
https://www.python.org/dev/peps/pep-0384/#structures

I can understand that it's annoying to not be able to use the same wheel package on Python 3.7 <= 3.7.4 and Python Python 3.7 >= 3.7.5, but I don't think that we can fix this issue.

I suggest to close it as "rejected".

One solution is to require to build your C extension on installation (don't ship wheel packages).

If you want to ship wheel package, another compromise would be to require Python 3.7.5 or newer, and check early the Python version.

[vstinner]

This happens on a custom C module (built via Cython) when using including <internal/pystate.h> with -DPy_BUILD_CORE. I'm not sure it'd happen otherwise.

By the way, I don't think that wheel packages are designed to C extensions accessing Python internals.

The internal C API is excluded from the stable ABI.

[jd]

If you want to ship wheel package, another compromise would be to require Python 3.7.5 or newer, and check early the Python version.

Yes, this is what I do now to avoid getting a segmentation fault.

I'm fine with closing this as wontfix or reject, your call. As you said, if it's not part of the stable ABI it makes sense.

The sad part if that my code does not access PyGC_Head at all. It's just a side effect when loading the code that makes it crash. :(

[vstinner]

The sad part if that my code does not access PyGC_Head at all. It's just a side effect when loading the code that makes it crash. :(

Python stores pointers to PyObject*. If your code doesn't access PyGC_Head directly, maybe it does acess it indirectly? I don't understand why you get a crash. It would help if you can provide more information about the crash. In which function does it happen? Try to get a gdb trace.

[mdickinson]

[Julien Danjou]

As you said, if it's not part of the stable ABI it makes sense.

I think this is a misunderstanding of what the stable ABI is for. I'm not sure whether this is spelled out explicitly anywhere, but my understanding was that ABI breakage shouldn't happen *at all* in bugfix releases, whether stable ABI or not.

The stable ABI is about feature releases within the Python 3.x series, not bugfix releases: that is, if your extension module limits itself to using the stable ABI, then a version of that module compiled for Python 3.7.x should still work for Python 3.8.x.

But lots of extension modules don't or can't restrict themselves to the stable ABI. For those modules, it should still be reasonable to expect that bugfix releases don't break the ABI: you should always be able to upgrade from Python 3.7.5 to Python 3.7.6 (for example) without having to recompile your extension modules.

Or am I misunderstanding?

[methane]

cpython/Include/cpython/objimpl.h

Line 89 in 95905ce

#define _Py_AS_GC(o) ((PyGC_Head *)(o)-1)

_Py_AS_GC and all APIs to get PyGC_Head from PyObject* are not only unstable, but also private.

I thought PyGC_Head must not be used directly from extensions, so we changed its layout.

Since Julien said "my code does not access PyGC_Head at all", I still think it was not real ABI breakage.

I suppose other serious bug (e.g. dungling pointer, missing refcount increment, etc) caused the segfault.