1

Hello,
I merged a new version of PEP 803, based on previous discussion (core, packaging, and some private conversations).
The result is online; also see the PR to see exactly what changed.

The main change is that a new stable ABI, abi3t, is treated as separate from the existing abi3. You can compile against abi3 like in 3.14; you can newly compile against abi3t (with additional API limitations – that is you need to change your source). And you can compile against both at the time, which has no downsides over compiling for only abi3t so it’s what we recommend tools to focus on, but the door is open to abi3t-only extensions having some benefit in the future.

This comes with a new “knob”: you (the user, or a build tool) define Py_TARGET_ABI3T to build against abi3t, in the same way you define Py_LIMITED_API to build against abi3. (The asymmetry in the names isn’t great, but “Limited API” is a confusing term and it’s best to start avoiding it.)
“In the same way” means that you define the macro to the lowest CPython version your extension should be compatible with.

With Py_GIL_DISABLED (and/or, on non-Windows splatforms, with headers for a free-threaded Python), defining Py_LIMITED_API will automatically define Py_TARGET_ABI3T (rather than cause a sompile error).

That’s the main change. Let me know what you think – but as always, check if the PEP doesn’t already address your questions & concerns.

6 Likes
2

As an infrequent user of any ABI, I am still quite confused, even though I like the simplicity of the orthogonal LIMITED and ABI3T options.

Could you prepare a matrix showing which combination of ABI macro settings (none, LIMITED, ABI3T, both) produces an extension compatible with which Python versions (and for >= 3.14, with the GIL and GIL-free builds)?

Separately, is the GIL setting selected by the extension module (or its build tools) or is it hard-coded in Python.h or something that includes?

1 Like
3

A single table would be too wide, so here’s two – one for what you get with given macro settings (and Python build); one for what the compatibility will be: PEP 803 – “abi3t”: Stable ABI for Free-Threaded Builds | peps.python.org
Let me know if you’d prefer a different format, or more rows.

On Windows, it’s selected by the user/tool (by defining Py_GIL_DISABLED).
Elsewhere, it’s hardcoded in pyconfig.h (included by Python.h).
(The PEP doesn’t change this. For more discussion, consider “Reply as a linked topic” in order to not derail the discussion here.)

1 Like
4

Would it make sense to introduce Py_TARGET_ABI3 at the same time? That way, Py_LIMITED_API could live on solely as a backwards compatibility alias, rather than as the only way of targeting the GIL-protected ABI.

3 Likes
5

The long-term plan is that the free-threaded build becomes default, and the non-free-threaded is eventually dropped.
Even if that takes a decade+, I don’t think this rename would be worth the churn.

2 Likes
6

I tried porting nanobind to abi3t (with _Py_OPAQUE_PYOBJECT) to see how it would go and ran into one problem.

nanobind uses a custom function object, which derives from PyVarObject. The variable number of entries is related to the number of function overloads (i.e., C++ functions with the same name but different argument type signatures).

With abi3t I can allocate a PyVarObject-derived object of a given size, and I can also query the number of entries via Py_SIZE(..). But it’s not clear to me how I can get a pointer to the N-th data entry of this function object. In the past, I did something like sizeof(PyVarObject) + sizeof(Payload) * index to compute an offset, but my understanding is that that isn’t allowed since extensions should not depend on sizeof(PyVarObject).

If that can be resolved then nanobind should be able to adopt this new abi3t compilation target.

2 Likes
7

With Py_GIL_DISABLED (and/or, on non-Windows platforms, with headers for a free-threaded Python), defining Py_LIMITED_API will automatically define Py_TARGET_ABI3T (rather than cause a compile error).

In the table PEP 803 – “abi3t”: Stable ABI for Free-Threaded Builds | peps.python.org two of the rows have “Compile on …” as “3.15+” without specifying “(GIL)” or “(FT)“. I’m a little confused.

There are six rows for 3.15. Maybe it would be worthwhile to list all eight possibilities: (GIL/FT) x Py_LIMITED_API x Py_TARGET_ABI3T

8

I think depends if you’ve set Py_TPFLAGS_ITEMS_AT_END. If you have then it’s just PyObject_GetItemData to get the first entry. And then sizeof(Entry) * N offset to get the Nth.

If you haven’t, then I think you use PyObject_GetTypeData to get the main data object. Then offset by the size of that struct to get into the per-item data.

The ITEMS_AT_END case is probably clearer, but I think there’s only a major difference if there’s inheritance involved (which I suspect there isn’t in your case?).

Edit: Found the other place you were discussing it Explore support for Python 3.15 limited API extensions · wjakob/nanobind · Discussion #1284 · GitHub where you correctly point out PyObject_GetItemData isn’t in the stable ABI. That suggests you need to to PyObject_GetTypeData plus the size of your fixed size struct and that you can’t handle inheritance.

9

@encukou do you happen to remember why you excluded PyObject_GetItemData from the stable ABI?

PEP 697 just says that it’s excluded, but I don’t see a reason.

10

Right. And PyObject_GetTypeData() (which is in the stable ABI) is also not usable here. For a PyVarObject argument, that function produces to the same offset as a for a PyObject argument. So I manually need to shift by sizeof(size_t) to account for the size field. This seems non-portable in a stable ABI extension that should not rely on how a PyVarObject is composed internally.

11

Yes, the stable ABI doesn’t really give you a way to define PyVarObject if PyObject is opaque.

3.12’s PEP 697 was meant for special purposes, not for defining all the classes of an extension is in abi3t. Like with critical sections, we need new stable ABI for that.
Thanks for flagging this; it looks like the next priority after critical sections.

12

The rows without the (GIL) or (FT) note are valid for either variant.

13

The following statement at the beginning of this discussion:

[When compiling on (FT)] defining Py_LIMITED_API will automatically define Py_TARGET_ABI3T

seems to conflict with the data in the table. The next-to-last line of the table shows defining only Py_LIMITED_API results in cp315-abi3t, but the last line of the table shows defining both Py_LIMITED_API and Py_TARGET_ABI3T results in cp315-abi3.abi3t.

Another thought, if you don’t mind: there’s three version numbers. I assume it will be supported to build, for example, cp315-abi3.abi3t on Python 3.16 if one sets both macros to 3.15. What happens if the two macros are set to different values? Maybe compiler error? Or, is the max of the two used?

14

Ah, right! Thanks; I’ll fix this in the next update.
Extensions built on 3.15+ (FT) with Py_LIMITED_API should be tagged cp315-abi3.abi3t.

That depends on your point of view:

  • conceptually: You need to honor the limitations each of the settings imposes; you receive the compatibility guarantees each one provides.
  • practically: CPython will set Py_LIMITED_API to the lower version, and use that. (This will be an implementation detail, subject to change.)
  • advice for build tools: Don’t do that. There’s no way to encode this in wheel tags, anyway.
15

I did some work this week to gather and solicit comments from ecosystem maintainers on this proposal. Petr just merged the PR adding the comments I found and received.

Petr also fixed the issue with the table @hpkfft noticed.

I added myself as a co-author. This is my first time showing up on a PEP :slight_smile:.

The PEP is now updated and you can read the new section here:

2 Likes
16

If Py_LIMITED_API=v1 and Py_TARGET_ABI3T=v2 and v1 != v2, I would guess that was not done deliberately, but rather happened because a build tool was misconfigured, or compiler flags were wrongly added, or entries in pyproject.toml were overwritten in CMakeLists.txt, etc. The build tool might encode its configuration in the wheel tag without realizing that the compiler downstream is seeing different macro definitions. I might suggest, as an implementation detail, that it would be more user friendly to “fail fast” and cause a compilation error if v1 != v2. If you agree, then maybe omit the phrase “or redefine” in the following:

If Py_TARGET_ABI3T=v is set, CPython may define or redefine Py_LIMITED_API as v.

Regardless, I would like to thank you all for your work on this PEP. It will be valuable to us. The cost for us is not so much building different wheels but rather in the quality assurance testing of each wheel. This limits how many wheels we’re able to support.

Another advantage to having this PEP implemented is that old versions of our FFT extension should continue to work on future releases of Python. Consider the case of our removing something (after a deprecation period) in the latest version of our extension. A user, having ignored the deprecation warning and relying on the removed feature, might well ask us to provide an old version of our extension on a new release of Python. With this PEP, our software version is decoupled from the Python software version. So, again, thank you.