Proposal: Officially Deprecate Support for Legacy (Single-Phase Init) Extension Modules

ngoldbaum · 2025-04-23T20:14:00.874Z

Eric Snow:

One follow-up question: do you think the effort to consolidate (and isolate) global state will help with support for running under the free-threading build?

Yes, it was necessary to finish shipping free-threaded support in NumPy 2.1 last year. Unless you’re asking generically for other libraries that you’re encouraging to do this. In which case, yes, identifying global module state is necessary for a C extension to declare free-threaded support, and if you’re doing that you might as well centralize it, if only to keep things organized. And you might find horrors like storing the state for a context manager in a C global variable, which works about as well as you might expect it in any non-trivial multithreaded code.

See MNT: Reorganize non-constant global statics into structs by ngoldbaum · Pull Request #26607 · numpy/numpy · GitHub if you’re curious about the code details in NumPy and want to dive into a thousand-line C diff

wjakob · 2025-04-24T00:48:53.622Z

Hi @eric.snow. nanobind maintainer here. A correction to what you said: multi-phase initialization is not a prerequiste for free-threading. Extensions with single phase initialization can opt in via PyUnstable_Module_SetGIL, and this is what e.g. nanobind currently does.

Conceptually, it would not be hard to adapt nanobind to multi-phase initialization. However, in the past I decided against doing this because the benefit seemed unclear. Please correct me if I’m wrong with the following: to my knowledge, the main point of multi-phase initialization is to encapsulate module state in a non-global way, which in turn permits the use of subinterpreters.

However, there are problems with subinterpreters for binding tools like nanobind or pybind11. To enable C++/Python interoperability, they rely on internal data structures that track the state of type and instance objects. Those data structures are global because types could be used by many different modules running in the same interpreter. Different extensions built with nanobind communicate with each other through this shared state.

Currently, nanobind caches a global pointer (static Internals *internals = ...) in all extensions, which points to this data structure following a costly lookup. A significant chunk of code relies on the ability to dereference this pointer. With subinterpreters, that is no longer possible, and it becomes rather complicated to access the module state depending on the context where it is needed. I am concerned that this will add significant costs to the fast path in nanobind. That’s why I never seriously pursued subinterpreter integration, which in turn made it logical to stick with single-phase initialization.

steve.dower · 2025-04-24T13:55:27.970Z

Wenzel Jakob:

Extensions with single phase initialization can opt in via PyUnstable_Module_SetGIL, and this is what e.g. nanobind currently does.

Eric’s made the point elsewhere that this is a temporary hack to allow immediate unblocking of some users. It’s not a permanent API, and we’d like to see users move onto the permanent API (multi-phase init).

The main point of multi-phase initialization is to future-proof your modules (or your user’s modules, in this case) against work that we do to improve imports. Without it, we are massively constrained on the runtime side of things, because we have no way to determine whether a module supports new features before it actively runs code.

With multi-phase init, we can safely load your module, discover that it doesn’t support (e.g.) subinterpreters, and wrap it up so that it doesn’t break. Without multi-phase init, we basically have to assume that we can’t trust the module to support anything new since around 2013. We don’t want to add new public API for each feature to activate a module’s support for features - we already have that public API (multi-phase init).

eric.snow · 2025-04-24T16:37:20.266Z

Thanks for the explanation. @steve.dower covered some of what I would have said. (For more discussion about the free-threading build and PyUnstable_Module_SetGIL(), check out my other thread: https://discuss.python.org/t/concerns-about-pyunstable-module-setgil/89374).

I do want to reiterate something I pointed out in an earlier comment, that implementing multi-phase init is not the same thing as implementing module isolation. Your explanation implies to me that you understand the two to be the same, which seems to be a fairly common misunderstanding. You can actually implement multi-phase init without the rest of module isolation. (See my earlier post in this thread and my post in the other thread for more info.)

Regarding subinterpreters, what you’ve written is very helpful. Thanks for taking the time to write it up. There isn’t a problem with saying “this module does not support use in multiple interpreters.” Multi-phase init supports that. That said, I’m sure the Python community will ultimately dictate how much users expect extension modules to support use in multiple interpreters.

Also note that we are seriously looking into eventually adjusting the C-API to make the CPython runtime context explicit, instead of relying on a thread-local variable (accessed through PyThreadState_Get()). This would mean nearly all functions would add a context argument (like HPy has), which would effectively be equivalent to an opaque PyThreadState. While this might not change the constraints on nanobind state management much (I’m not sufficiently familiar with nanobind to say), I expect such a change would have some impact on those constraints.

wjakob · 2025-04-25T04:17:14.005Z

@eric.snow @steve.dower Thank you for the explanation. This makes sense to me now. I will switch to multi-phase init for the next version of nanobind. I am not involved in pybind11 development anymore, but there is an open PR for subinterpreters+multi-phase initialization that is also likely to be merged soon.

I was wondering about two related points:

AFAIK both free-threading and subinterpreters are solutions to improve parallelism. Are they exclusive, or will there will be python builds with both free-threading and subinterpeters?
The need to distribute gigabytes of binary wheels for many different platforms and Python minor versions has been a pain point in the past, especially for bigger binding efforts. nanobind has a feature to make stable ABI extensions, which is possible since Python 3.12. However, with work on improving parallelization in Python, my impression is that there is no concept of a stable ABI (at least, that is definitely the case for free-threaded Python). What is the status of subinterpreters on this point?

da-woods · 2025-04-25T06:12:06.473Z

Wenzel Jakob:

AFAIK both free-threading and subinterpreters are solutions to improve parallelism. Are they exclusive, or will there will be python builds with both free-threading and subinterpeters?

I believe that subinterpeters isn’t a separate build of Python. It’s just in every build of Python including the free-threading one. So not exclusive.

Wenzel Jakob:

[…] stable ABI extensions […] What is the status of subinterpreters on this point?

Stable ABI is compatible with sub-interpreters I believe. From an extension author’s point of view you “just” have to mark the extension as compatible with sub-interpreters (which you do via moduledef slots) and make sure that all global state is on the module state structure. The latter is the hard bit, but is all in the stable ABI.

da-woods · 2025-04-25T06:49:36.941Z

FWIW pybind11 does look to support multi-phase initialization now as of about 3 weeks ago https://github.com/pybind/pybind11/pull/5574

steve.dower · 2025-04-25T09:14:49.034Z

Wenzel Jakob:

However, with work on improving parallelization in Python, my impression is that there is no concept of a stable ABI (at least, that is definitely the case for free-threaded Python). What is the status of subinterpreters on this point?

As Da Woods said, subinterpreters are already in the main release, so it’s as stable as the Stable ABI.

Free-threaded is an experimental build, because we don’t yet know exactly what shape things will have to be in order to work. Expect some breaking changes, or potentially an entirely new stable ABI (we’re discussing elsewhere whether to move to abi4 for all builds, to add a new parallel abi3t just for free-threaded, or just add a set of build-time options to choose which parts are stable). There’s a proposal that free-threaded is stable enough now that we can commit to its ABI, and so if that’s accepted then the new stable ABI can be determined, but until then we need to keep the flexibility to avoid designing in flaws and being committed to them for a long period of time.

eendebakpt · 2025-04-25T20:08:27.897Z

SWIG also uses single-phase init when Python extensions are created. A feature request on the github issue tracker has been created Support Python multi-phase init · Issue #3168 · swig/swig · GitHub.

da-woods · 2025-04-26T11:30:04.029Z

I’ve just remembered one place where single-phase initialization may be useful for some Cython users.

With single-phase initialization its possible to import a module directly by calling PyInit_module_name which is useful when you want to build all your code into one big binary rather than making each module a separate .so file as Python expects.

We try to encourage people to use PyImport_AppendInittab but that only works if you can set everything up ahead of time before the interpreter is initialized which not everyone can.

This isn’t a use-case that we try to encourage. But I know it’s something which some people do.

(I should also say, I have worked out and documented how to do the same thing with PyModule_FromDefAndSpec and PyModule_ExecDef but it’s pretty verbose and I’m honestly not convinced it’s less hacky)

eric.snow · 2025-04-28T17:46:37.274Z

Da Woods:

With single-phase initialization its possible to import a module directly by calling PyInit_module_name which is useful when you want to build all your code into one big binary rather than making each module a separate .so file as Python expects.

That should work almost just as well with multi-phase init, with some extra steps involved. We do this in the CPython test suite:

https://github.com/python/cpython/blob/main/Modules/_testimportmultiple.c
https://github.com/python/cpython/blob/main/Modules/_testmultiphase.c

For a specific example, see PyInit__test_non_isolated() in Modules/_testmultiphase.c. It is defined in the same file with a bunch of other modules. We load it in a test with the following helper code:

def import_extension_from_file(modname, filename, *, put_in_sys_modules=True):
    loader = importlib.machinery.ExtensionFileLoader(modname, filename)
    spec = importlib.util.spec_from_loader(modname, loader)
    module = importlib.util.module_from_spec(spec)
    loader.exec_module(module)
    if put_in_sys_modules:
        sys.modules[modname] = module
    return module

Would it be worth adding a helper in importlib.util or as a method on ExtensionFileLoader, and/or a helper in the C-API?

Da Woods:

(I should also say, I have worked out and documented how to do the same thing with PyModule_FromDefAndSpec and PyModule_ExecDef but it’s pretty verbose and I’m honestly not convinced it’s less hacky)

This implies to me helpers would be worth adding.

FWIW, the approach of the helper I copied above is definitely the correct way. If that’s how you implemented it using PyModule_FromDefAndSpec() and PyModule_ExecDef() then that wasn’t hacky. The fact that you are uncertain tells me we have a deficiency in the docs and C-API.

da-woods · 2025-04-28T18:00:23.613Z

Eric Snow:

FWIW, the approach of the helper I copied above is definitely the correct way. If that’s how you implemented it using PyModule_FromDefAndSpec() and PyModule_ExecDef() then that wasn’t hacky.

It’s not quite the same. My approach is Issues when embedding — Cython Guidelines documentation

I think the main difference is that I bypass using a loader because the use-case is that the module doesn’t exist as a separate file - it’s just built into the main program.

neonene · 2025-05-24T00:47:13.117Z

Eric Snow:

the Py_mod_multiple_interpreters slot[3]
set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED [4]

The implementation would be more trivial if Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED (NULL) becomes the default.

Now that modern subinterpreters support per-interpreter GIL, the Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED option seems too special to be implicitly chosen? I suppose no authors are currently omitting the slot. If that is not wrong, the default could be switched.

encukou · 2025-05-26T08:12:16.741Z

Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED is awkward because it’s the backwards-compatible option. It’s been the default since 2013, when PEP 489 introduced PyModuleDef_Slot itself – long before the Py_mod_multiple_interpreters slot was added. See the Subinterpreters and Interpreter Reloading section in PEP 489.

IMO, the isolation “opt-out” pattern is a bit more general (and more explicit) than the Py_mod_multiple_interpreters slot. But unfortunately, it also blocks more scenarios that can be unsafe:

Multiple Py_Initialize/Py_Finalize cycles: if you have a static PyObject* pointer, it can be left dangling in that situation.
What NumPy tests call “full reimport” – del sys.modules[...]; import .... In this case, the multi-phase behaviour (real full reimport) is very different from single-phase (reused __dict__ contents).

neonene · 2025-05-26T15:52:18.886Z

Petr Viktorin:

It’s been the default since 2013, when PEP 489 introduced PyModuleDef_Slot itself – long before the Py_mod_multiple_interpreters slot was added. See the Subinterpreters and Interpreter Reloading section in PEP 489.

I can give an (expanded) interpretation to the section:

“Multi-phase init modules are not expected to support subinterpreters until full isolation is achieved.”

In that case, the multiple-interpreters support sounds like a non-default option at least when introducing. I’d prefer the reconsideration if possible.

ngoldbaum · 2025-05-26T16:14:00.114Z

Following up on NumPy, @AA-Turner has been busy and at least has an outline towards converting NumPy’s extension modules to multi-phase init!

github.com/numpy/numpy

ENH: Use multi-phase initialisation (PEP 489) for extension modules

opened 07:28PM - 21 May 25 UTC

AA-Turner

### Proposed new feature or change: This is intended to be a tracking issue for… a series of PRs I will shortly submit. Multi-phase initialisation was introduced in [PEP 489](https://peps.python.org/pep-0489/) for Python 3.5. It replaces the previous 'single-phase' mechanism by splitting the module creation process into creation and execution phases. Quoting from [the documentation](https://docs.python.org/3/c-api/module.html#multi-phase-initialization): > Extension modules created this way behave more like Python modules: the initialization is split between the creation phase, when the module object is created, and the execution phase, when it is populated. The distinction is similar to the `__new__()` and `__init__()` methods of classes. This is also the stable mechanism to declare compatibility with both free-threading and subinterpreters (though note that **this PR does not attempt to add support for subinterpreters**). A common point of confusion/conflation is multi-phase initialisation and module isolation. As noted on the [Python Discourse](https://discuss.python.org/t/89374) forum, "*Implementing extension module isolation is a very good thing, but if a module uses static types or (C) global variables then the effort to isolate becomes non-trivial.*". This issue only seeks to implement multi-phase initialisation, generally following the guide that Eric set out in that thread: 1. move the content of the module’s init function to a corresponding “module exec” function (dropping the call to create the module object) 2. set the module def’s [`m_slots`](https://docs.python.org/3/c-api/module.html#c.PyModuleDef.m_slots) field to an array with: * a `Py_mod_exec` slot, set to the new module exec function * a `Py_mod_multiple_interpreters` slot, set to `Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED` * to try free-threading, a `Py_mod_gil` slot, set to `Py_MOD_GIL_NOT_USED ` 3. set `def.m_size` to 0 (if negative) 4. update the module init function to only `return PyModuleDef_Init(def);` for the corresponding module def cc @ngoldbaum xref: * https://github.com/numpy/numpy/issues/26157#issuecomment-2079486111 * https://github.com/numpy/numpy/issues/28271#issuecomment-2634857326 * https://github.com/numpy/numpy/issues/24755 * https://github.com/numpy/numpy/issues/11925#issuecomment-421082106 * https://github.com/numpy/numpy/issues/665 A PRs: * #29022 * #29023 * #29024 * #29025 * #29026 * #29027 * #29028 * #29029 * #29030 * #29031 * TODO: simd_dispatch * TODO: f2py

AA-Turner · 2025-05-26T16:27:18.017Z

I’ve also just seen that SWIG has a draft PR to support multi-phase at Python: Use multi-phase initialization by jschueller · Pull Request #3184 · swig/swig · GitHub (by @jschueller)

A

encukou · 2025-05-26T16:28:42.309Z

The section starts with:

Extensions using the new initialization scheme are expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly, avoiding the issues mentioned in Python documentation.

I’m not sure how you can interpret that as multi-phase modules not being expected to support subinterpreters.

Anyway, the PEP is history; current docs have been saying this since 3.5:

All modules created using multi-phase initialization are expected to support sub-interpreters. Making sure multiple modules are independent is typically enough to achieve this.

encukou · 2025-05-27T06:44:53.088Z

Petr Viktorin:

Anyway, the PEP is history; current docs have been saying this since 3.5:

I opened PR #134775 to give more info here.

encukou · 2025-05-29T11:30:12.006Z

For the record: PR #134764 proposes to document single-phase init as legacy.
I think that matches the consensus here, but, I’m obviously biased.

The only reason to use single-phase init should be backwards compatibility. It can be a very strong reason, of course.