per-interpreter: memory allocators

[ericsnowcurrently]

[maybe split into more granular tasks]

1. move the global state back to Include/internal (see below)
2. move PyRuntimeState.obj to PyInterpreterState.obj
3. move PyRuntimeState.mem to PyInterpreterState.mem

Notable concerns:

• very delicate code
• allocators needed before main interpreter created (?)
• keep a global allocator?

---

The global state for this was moved back out of Include/internal to fix a bug. That PR encapsulates the relevant global state and C-API pretty well.

[qizzzh]

Can we have per interpreter heap and allocators and a flag telling whether we are in the main interpreter? The overhead doesn't sounds terrible.

[ericsnowcurrently]

"per interpreter allocators" is exactly what this issue is for.

What do you mean by "per interpreter heap"? The C heap wouldn't make sense so I'm guessing you're talking about the Python "heap". If so, that's pretty much unrelated to what we're working on here. Consider starting a thread on the python-ideas or python-dev mailing lists.

Where are you looking for a flag about the main interpreter? In the C code all the info is there already. From Python the information is exposed in PEP 554.

[vstinner]

A workaround is to disable pymalloc. It can even be done by setting PYTHONMALLOC env var to "malloc" for example. libc malloc() is thread safe.

[ericsnowcurrently]

@vstinner, a workaround for what? per-interpreter allocators or the bug for which we rolled back moving the allocators into PyRuntimeState?

[vstinner]

@ericsnowcurrently: "@vstinner, a workaround for what?"

pymalloc is not thread safe. If you want to experiment subinterpreters running in parallel in different threads, you can force the usage of the libc malloc()/free() by setting PYTHONMALLOC env var to "malloc".

[vstinner]

A workaround is to disable pymalloc. It can even be done by setting PYTHONMALLOC env var to "malloc" for example. libc malloc() is thread safe.

I proposed the following issue and PR to implement this workaround as a configure build option:

• https://bugs.python.org/issue40514
• bpo-40514: Add --with-experimental-isolated-subinterpreters python/cpython#19926