Stable ABI/Limited API for free-threaded builds

encukou · 2025-06-26T09:38:09.781Z

Steve Dower:

Even if we did this, we can’t change the layout of PyObject until whenever we decide that limited API modules built with 3.14 are no longer supported. There isn’t an agreed upon definition of this deprecation timeline (that I’m aware of), so it seems likely that we’ll need a hard break migration at some point before it expires anyway. May as well be planning for that (and supporting it in parallel as much as we can) rather than trying to patch over design decisions from the past.

That hard break is called “Phase III”. No timeline, but it seems it could be pretty soon.

I agree about making the transition as painless as possible. That’s why I propose both API compatibility (no source changes → separate abi3 & abi3t builds) and ABI compatibility (PyModExport, opaque PyObject → abi3.abi3t builds).
We want extensions to switch to the latter (so that we can change PyObject in the future), but I think we need a window where we support both – both for extension authors to switch, and for CPython to react to feedback.

rgommers · 2025-06-26T10:57:18.349Z

Petr Viktorin:

We want extension to switch to the latter (so that we can change PyObject in the future), but I think we need a window where we support both – both for extension authors to switch, and for CPython to react to feedback.

This seems like a good strategy to me. Based on experience with recent large-scale migration efforts - NumPy 2.0 and free-threading - a one year migration window is likely too tight. The inertia of a lot of projects having to do releases is very large, even if no source code changes are needed. With a two year or three year window, it should be possible to do it in a low-disruption manner though.

steve.dower · 2025-06-26T13:29:38.710Z

Petr Viktorin:

That’s why I propose both API compatibility (no source changes → separate abi3 & abi3t builds) and ABI compatibility (PyModExport, opaque PyObject → abi3.abi3t builds).

If “no source changes” is true, can we make the non-freethreaded build also support abi3t? That way only one build is required, at least for ~3.15 onwards, while builds made prior to 3.15 (and for versions prior to 3.15) continue to work on non-freethreaded builds for a while.

encukou · 2025-06-26T13:54:13.292Z

Steve Dower:

If “no source changes” is true, can we make the non-freethreaded build also support abi3t?

For extensions that do not use the PyObject layout, yes!
But for most extensions, avoiding the PyObject layout means a source change – see the PEP.

steve.dower · 2025-06-26T14:04:02.594Z

Yeah, the end result is nearly the same, but I think there’s a difference in what package maintainers have to worry about.

Basically, if all 3.15 builds support abi3t, whether free-threaded or not, you only need to publish one “universal” package for 3.15 and later. Rather than having to build twice and publish two packages (how your proposal currently reads).

If you want universal builds for 3.14 and earlier and also 3.15 free-threaded and later, you need to do both abi3 and abi3t.

If you don’t want to do any builds at all, abi3 continues to work with non-freethreaded 3.15 and continues to not work with freethreaded.

So essentially, we can do a new ABI that works regardless of freethreading or not, it first works with 3.15, and we’ll keep the old ABI around as long as we can (which I suspect may not be long, but it’ll be longer than dropping it immediately). So set the transition point at a Python version, rather than at a build option.

(This should seem perfectly in line with my preference for calling it abi4 rather than 3t, and maybe my explanation makes more sense if you substitute 4 in there.)

encukou · 2025-06-26T14:39:37.906Z

Steve Dower:

Basically, if all 3.15 builds support abi3t, whether free-threaded or not, you only need to publish one “universal” package for 3.15 and later. Rather than having to build twice and publish two packages (how your proposal currently reads).

My proposal calls that abi3.abi3t.
Yours would call that cp315-abi3, or abi4, if i understand correctly.
Implementation-wise, it’s about setting defaults, (dis)allowing options, and naming things. The options are open, but as far as I can see, they all need opaque PyObject (either with Py_OPAQUE_PYOBJECT or implicitly) and PEP 793.

steve.dower · 2025-06-26T14:47:13.944Z

Petr Viktorin:

My proposal calls that abi3.abi3t.

Except it isn’t, really, because it’s not going to work with Python 3.14 or earlier. And having abi3 in the tag means that it should. So really, your proposal is cp315-abi3.abi3t, whereas mine is just cp315-abi3t (or cp315-abi4, since the t would imply free-threaded builds which aren’t a requirement under my proposal).^[1]

The big problem we have is that abi3 is meant to be forward and backward compatible, but we gave up on that, and so in practice it’s just “cp3X or greater”. Adding a second ABI and claiming it’s interchangeable with the existing one (the meaning of abi3.abi3t) only really makes it more confusing. (What does cp314-abi3.abi3t mean, for example, when abi3t isn’t available there and abi3 is different from the runtimes where abi3t exists?)

Fiddling at the edges with minimal changes and new compile time options doesn’t solve the problem that we ultimately need a clean break, and need a compatible way to add new interfaces later on. Adding things that create complexity, can’t be easily removed, and don’t solve the problem will only make things worse.

And cp315-abi3 would still be valid for non-freethreaded builds. ↩︎

encukou · 2025-06-26T15:00:28.615Z

Steve Dower:

What does cp314-abi3.abi3t mean, for example, when abi3t isn’t available there and abi3 is different from the runtimes where abi3t exists?

It means you use the limited API (abi3), treat PyObject as opaque (abi3.abi3t), and don’t require any 3.15+ APIs (cp314).
Which is hard to pull off, but possible.

Steve Dower:

Adding things that create complexity, can’t be easily removed, and don’t solve the problem will only make things worse.

Please be explicit about the “things” you mean.
I think that at this point i need to ask you for a more complete counter-proposal; this discussion is going in circles.

steve.dower · 2025-06-26T15:14:19.247Z

Petr Viktorin:

I think that at this point i need to ask you for a more complete counter-proposal; this discussion is going in circles.

You’re already on top of most of the details, it’s just the labelling and the wheel/so tags we’re debating.

So my complete proposal is that we define abi4 with all the technical changes needed to support free-threaded builds and future forwards+backwards compatible expansion, implement it in 3.15 in all build configurations, freeze abi3 and only keep it in in build configurations where it works (i.e. not free-threaded), marked as deprecated and discouraged until the release we have to completely drop it.^[1] Defining Py_LIMITED_API omits all non-abi4 members from the headers, with essential fallbacks^[2] for Py_LIMITED_API<0x04000000. Existing abi3 releases continue to work everywhere they would work already, and new releases can be made using abi3 by package developers who refuse to update their code, and those who will update can fix compiler errors until they fit into abi4 (perhaps involving their own #if Py_LIMITED_API+0<0x04000000 checks, depending on which versions they care to support).

I’ll admit, this only really makes sense if we think free-threaded is inevitable. The SC sure sounds that way, so I think it’s responsible for us to assume it will be and we have to provide a reasonable way for package developers to handle it - hence, people who want to support it have to make code changes anyway (and for far more complicated problems than compiler errors!), and so migrating them to abi4 is a simplifying step. But if those code changes aren’t needed (because we give up on free-threading) then migrating to abi4 is pure pain, and I wouldn’t support it.

In any case, I don’t support compiler flags or ABI flags that indicate “I never rely on the members or shape of PyObject”. Either it’s allowed by the ABI or it isn’t, and if it isn’t allowed then it should just be a new ABI so we can properly change what we need to change to enable it. We can even require use of a new module export with a new ABI, which greatly simplifies that whole exercise^[3].

Because we’re actually changing PyObject, not merely making it opaque. ↩︎
Such as ob_refcnt and friends. ↩︎
To me, the challenge there is the migration process, not the technicalities of how it looks/works. No matter how perfect or imperfect, people won’t want to change to it, so we need to offer a better reason than “we assumed you wrote bad code before”. ↩︎

encukou · 2025-06-26T16:18:01.003Z

Sounds like bikeshedding to me! Yay!
If PEP 793 is accepted, I’ll

add a private macro (_Py_OPAQUE_PYOBJECT) for easier experimentation, with a promise to remove it well before 3.15 RC, and
summarize the options & open a thread specifically for the tags & names.

Does that sound fair?

steve.dower · 2025-06-26T16:35:30.419Z

I can’t really argue with a private macro, but I still think it’s an unnecessary burden. If there’s experimentation you want to do, you’ve already got a fork with that change, and if you’re trying to distribute the work to the package maintainers who install our alphas/betas then I think we can be a bit more decisive by then (and instead ask them to use something that won’t be removed, but might be modified based on feedback).

I don’t think we need a new thread, unless everyone’s muted this one because it’s just been us two for the last 50 messages. The core question is do we want a new free-threading-specific Stable ABI, or a new Stable ABI that works for all build configurations (noting that neither will be compatible with the existing ABI). It still sounds like you’re committed to the former and me to the latter.

The only reason I care about abi3t vs abi4 is because of that core question, and if everyone says they want a non-freethreaded ABI and a freethreaded ABI, then I’m not going to argue about the name further. But I’ll be surprised if people want to formalise two ABIs for the rest of time rather than only having a single ABI for 3.15+.^[1]

Not counting the then legacy abi3 that stays as long as it can. ↩︎

h-vetinari · 2025-06-27T10:27:35.240Z

+1 for a single ABI longterm^[1], even if it means some transition pain (the plans here already provide ample mitigation for that pain IMO; i.e. roll cut over to new ABI with 3.x, keep the old abi3 around for 2-3 years, then drop it).

Steve Dower:

because it’s just been us two for the last 50 messages

image256×185 1.36 KB

You’re too hard on yourself.

assuming freethreading stays, which seems like a foregone conclusion though ↩︎

pf_moore · 2025-06-27T11:01:09.241Z

H. Vetinari:

+1 for a single ABI longterm

Same from me. I don’t build packages using the C API/ABI myself, but I believe that a single stable ABI (with a well defined versioning scheme) is the best approach for packaging such projects.

encukou · 2025-06-27T15:46:19.724Z

Steve Dower:

The core question is do we want a new free-threading-specific Stable ABI, or a new Stable ABI that works for all build configurations (noting that neither will be compatible with the existing ABI). It still sounds like you’re committed to the former and me to the latter.

I want both. One can be API-compatible; the latter can’t^[1].

In my proposal, in the long term there’s one single ABI, called abi3.abi3t. (Which can in the long long term be shortened to abi3t, or abi4 if we make a breaking change. Or we can just call it abi4 now, if the spelling is important.)

Steve Dower:

I’ll admit, [abi4] only really makes sense if we think free-threaded is inevitable. The SC sure sounds that way, so I think it’s responsible for us to assume it will be and we have to provide a reasonable way for package developers to handle it - hence, people who want to support it have to make code changes anyway (and for far more complicated problems than compiler errors!)

There might also people who do not care about supporting free-threading, and are OK with the runtime warning that the GIL was enabled (which is the default that happens if you don’t add Py_MOD_GIL_NOT_USED to a module).

Does your proposal support Py_LIMITED_API=0x030f0000? (Or the other way: are those users locked out of other 3.15 additions, like PySys_GetAttr)
Does your proposal support adding new features in 3.16? (Would the cpp flag for cp315-abi4 be Py_LIMITED_API=0x040f0000?)
Or do we need forward-compatible slots in 3.15?

(practically) ↩︎

steve.dower · 2025-06-27T16:04:50.496Z

Petr Viktorin:

There might also people who do not care about supporting free-threading, and are OK with the runtime warning that the GIL was enabled (which is the default that happens if you don’t add Py_MOD_GIL_NOT_USED to a module).

The problem here is these are two different groups of people. Package maintainers choose whether or not to impose the runtime warning on application developers, and we’re the ones in the middle who have to decide whether that’s okay or not.

Petr Viktorin:

Does your proposal support Py_LIMITED_API=0x030f0000? (Or the other way: are those users locked out of other 3.15 additions, like PySys_GetAttr)
Does your proposal support adding new features in 3.16?

I mean, I guess we can keep adding stuff to abi3 after we deprecate it, if that’s what we need/want to do. But people can only use those new things by making source code changes, which goes against your argument that people won’t have to make source code changes.

Petr Viktorin:

(Would the cpp flag for cp315-abi4 be Py_LIMITED_API=0x040f0000?)

No, because I really want to stop the idea that the stable ABI changes with each release. Hence the forward-compatible slots, so that new interfaces can be added that way (which moves knowledge of the slots into extension compile time, so it can be aware of the latest available ones, but implementation belongs to the runtime, and so it can safely reject slots it doesn’t know about - callers just have to have their own fallbacks for when a new feature is missing, which might be to raise an error, but hopefully will be to just use a slower/alternate codepath).

If/when we one day need abi4_1, then you’d use Py_LIMITED_API=0x04010000 to select it. But I’m hoping that never comes.

So yes, any implementation that supports abi4 would need the forward-compatible slots. They don’t have to be available in abi3 though (but of course are available to anyone not using the limited API).

encukou · 2025-08-01T11:16:24.705Z

H. Vetinari:

+1 for a single ABI longterm[1], even if it means some transition pain

This was also a common opinion when I asked around at EuroPython.

And making the ABI backwards-compatible is not as important as I thought. One can always build extra wheels with an older Python.
Also, an API change seems less problematic than I thought. Since extensions opt in to new versions of limited API, it can be done there.

So, my updated current plan is option 2 above:

Version 3.15 of the limited API makes `PyObject` opaque

The following API is removed:

PyObject_HEAD
_PyObject_EXTRA_INIT
PyObject_HEAD_INIT
PyObject_VAR_HEAD
struct _object (i.e. PyObject)
struct PyVarObject
PyModuleDef_Base (made opaque)
PyModuleDef_HEAD_INIT
PyModuleDef (made opaque)
Py_SIZE (can be added back as a function)
Py_SET_TYPE (ditto)
Py_SET_SIZE (ditto)

(You can test the effects in current main, by defining _Py_OPAQUE_PYOBJECT before including <Python.h>. Note that this macro will be either removed or renamed for 3.15.0.)

Module creation API is left in, but becomes unusable in practice (since
you can’t create the input for these functions):

PyModuleDef_Init
PyModule_Create, PyModule_Create2
PyModule_FromDefAndSpec, PyModule_FromDefAndSpec2

New export hook

To allow defining modules with opaque PyObject, we add the PyModExport
hook from PEP 793 (now submitted to the SC).

Wheel tags

There are two main options:

Wheels that use the new version of the ABI use the abi3.abi3t ABI tag.

This adds an ugly new tag.
Tools learn that cp315 and above + ABI tag abi3 is compatible with free-threading.

This avoids a new tag, but is much less obvious.

For the long term: tools that don’t support 3.14 any more may only use option 2, and ignore abi3t.

Runtime ABI checks

It still makes sense to me that build/install tools are responsible for
not putting incompatible extensions in site-packages – the metadata they can use is much richer.
But, we can provide some checks for when the tools get it wrong (or you’re not
using a tool that follows the latest spect).

Add an ABI checking slot is added, and made mandatory if PyModExport is used (that is: for stable ABI 3.15+).
This will prevent new extensions (stable ABI 3.15+) from being loaded in incompatible interpreters.
In free-threaded builds, PyModuleDef_Init should reject extensions compiled for the pre-free-threading ABI.
(PyModuleDef_Init is also called from PyModule_Create* and PyModule_FromDefAndSpec*`, so this covers single-phase init too.)

I’m testing a prototype that looks at a certain bit pattern which appears in PyModuleDef of free-threaded builds but not in PyModuleDef_HEAD_INIT of the current abi3.
This is fragile, and the check might need to be removed as PyObject internals change further. But, hopefully it can stay in for the transition period toward free-threading.

What’s missing is preventing existing free-threading extensions (cp314t) being loaded in gil-ful builds of Python, but, that’s out of scope of this thread.

Sunsetting; ABI support windows

The current stable ABI (3.14 & below) will, necessarily, become less useful when free-threading builds become default (phase III of the rollout), and unusable after the GIL builds are removed entirely.

We might also plan to regularly deprecate older versions of stable ABI after some time.
With the ABI checking slot, we can start emitting deprecation warnings as soon as we agree on a policy; there’s no rush to start the discussions now.

New API

[help welcome]

Add APIs like PyMutex and PyCriticalSection to stable ABI, so extensions with Py_MOD_GIL_NOT_USED are practical.

colesbury · 2025-08-01T19:28:02.966Z

This is my favorite version of the proposal so far.

encukou:

Tools learn that cp315 and above + ABI tag abi3 is compatible with free-threading.This avoids a new tag, but is much less obvious.

I think either naming scheme can be made to work, but the second option, designating that cp315-abi3 is compatible with free threading, seems much nicer to me:

It only requires a small change to tooling similar to the change we made two years ago.
The failure mode for old tooling (like pip) is not so bad: they may refuse to load cp315-abi3 packages in 3.15t builds, but they will still load them in non-free-threaded 3.15 Python builds.

wills721 · 2026-01-29T19:26:44.119Z

Late reply but has anything been done here? Python 3.14.2 is generally available. You can build Python 3.14.2 with –disable-gil. However, if you rely on awscrt it will not build due to this issue. Is there some way to make this work?

ngoldbaum · 2026-01-29T19:30:57.268Z

Python 3.14 free-threaded builds don’t support the limited API. Hopefully 3.15 will support a version of the limited API that is compatible with both builds.

For now, extensions that hard-code a need for the limited API in their builds will need to disable that for the free-threaded build.

You should perhaps try the suggestion here.

wills721 · 2026-01-29T19:59:25.505Z

Yes, I’ve seen that. The change for 3.14 there is not yet merged.