ABI incompatibility between 3.5.0/3.5.1 and 3.5.2+

[aldanor]

Here's the problem: if you build a pybind11 when using 3.5.2 headers, it will reference this symbol:

U _PyThreadState_UncheckedGet

because of how get_thread_state_unchecked is implemented. If you try to import this module with 3.5.0/3.5.1 version of Python, you will first get a linkage error and then most likely a segfault:

ImportError: foo.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _PyThreadState_UncheckedGet

Now, this is not very cool, since ABI compatibility between micro versions is something to be expected by default. It gets much worse if you use conda: it differentiates package versions by the major+minor numbers in the version only, so you have packages like foo-py34-0.1.0 and foo-py35-0.1.0 and it's implicitly assumed they should work well for all micro versions of the interpreter.

[wjakob]

I am aware of the issue and don't have a good solution for this. It was actually a royal pain to get the threading stuff to work for every possible version of Python, linking issues aside.

[aldanor]

Looking at cpython source for 3.5 branch:

#define GET_TSTATE() \
    ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))

PyThreadState *_PyThreadState_UncheckedGet(void) {
    return GET_TSTATE();
}

get_thread_state_unchecked() in pybind11:

inline PyThreadState *get_thread_state_unchecked() {
#if   PY_VERSION_HEX < 0x03000000
    return _PyThreadState_Current;
#elif PY_VERSION_HEX < 0x03050000
    return (PyThreadState*) _Py_atomic_load_relaxed(&_PyThreadState_Current);
#elif PY_VERSION_HEX < 0x03050200
    return (PyThreadState*) _PyThreadState_Current.value;
#else
    return _PyThreadState_UncheckedGet();
#endif
}

So, in 3.5.2+ you're doing exactly the same thing as in this case: PY_VERSION_HEX < 0x03050000 (3.0-3.4), however you are also introducing an extra symbol which may (and will) cause linkage errors?

[wjakob]

The curious bunch of #ifdefs is to work around a regression in Python (https://bugs.python.org/issue26154) where one of the symbols was not exported. So just removing one of the ifdefs would cause the builds to break on that version of Python.

[aldanor]

Would this work though (preserving the 3.5.1 hack but avoiding pulling in _PyThreadState_UncheckedGet on 3.5.* to avoid ABI incompatibility)?

inline PyThreadState *get_thread_state_unchecked() {
#if   PY_VERSION_HEX < 0x03000000
    return _PyThreadState_Current;
#elif PY_VERSION_HEX < 0x03050000
    return (PyThreadState*) _Py_atomic_load_relaxed(&_PyThreadState_Current);
#elif PY_VERSION_HEX < 0x03050200
    return (PyThreadState*) _PyThreadState_Current.value;
#elif PY_VERSION_HEX < 0x03060000
    return (PyThreadState*) _Py_atomic_load_relaxed(&_PyThreadState_Current);
#else
    return _PyThreadState_UncheckedGet();
#endif
}

[wjakob]

Aha -- that seems reasonable. Did you test it on 3.5.2? If so, feel free to submit a PR.

[wjakob]

I'm not totally sure if this solves the fundamental problem though: what if Python is compiled on 3.5.1? In that case, will it run on 3.5.0 and 3.5.2+?

[aldanor]

Yea, it doesn't solve the problem completely unless you start adding runtime checks I presume (would a runtime if-else only on 3.5.1 solve it?)....

But at least now the binaries built on 3.5.2/3.5.3/... should not reference a newly introduced symbol in the middle of 3.5 cycle, so that's already something. There's no change of functionality as opposed to what's done right now, I guess.

Ideally, this should be tested on a full grid of 3.5 releases of course.

[wjakob]

I'm hesitant to make further changes here unless the problem is really solved. In this case the Python guys screwed up, so there is only so much one can do.. An undefined symbol error is a reasonable error IMHO (as opposed to silent corruption of data structures or something like that).

[aldanor]

Yea, there's only so much you can do (without checking for Python version at runtime or something like that) so I'd have to concur although it's pretty sad. Maybe a few lines in the docs warning about ABI incompatibility?

[huncut2016]

I have the same issue.

OS: Linux 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Python version:Python 3.8.10.

Code:
https://github.com/av8ramit/pybind_example

When I try to run bazel test ..., then the code will compile, but when it's running I get this error: undefined symbol: _PyThreadState_Current. My friend uses the same configurations, but with a different kernel.