Repository navigation
Restarting the interpreter causes UB on 3.10.0a4 #87048
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-01-12.10:34:04.768>
created_at = <Date 2021-01-10.13:00:18.631>
labels = ['interpreter-core', '3.10']
title = 'Restarting the interpreter causes UB on 3.10.0a4'
updated_at = <Date 2021-02-04.22:08:11.618>
user = 'https://github.com/YannickJadoul'bugs.python.org fields:
activity = <Date 2021-02-04.22:08:11.618>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-12.10:34:04.768>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2021-01-10.13:00:18.631>
creator = 'YannickJadoul'
dependencies = []
files = []
hgrepos = []
issue_num = 42882
keywords = ['patch']
message_count = 8.0
messages = ['384761', '384850', '384855', '384903', '384904', '384910', '384912', '386499']
nosy_count = 3.0
nosy_names = ['vstinner', 'YannickJadoul', 'kj']
pr_nums = ['24193', '24198', '24440']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42882'
versions = ['Python 3.10']
Activity
YannickJadoul commentedon Jan 10, 2021
Issue detected in the embedding tests of pybind11, running on the latest alpha of 3.10: pybind/pybind11#2774
I have reduced the weird issue/crash to a minimal reproducer, which consistently reproduces the crash on my Linux machine:
Removing the two lines in the middle that restart the interpreter makes the example work.
I've also bisected CPython to find the issue (3.10.0a3 is fine, 3.10.0a4 is not), and arrived at #20058 (ba3d67c being the first commit that breaks the example above). But I am not entirely sure where to start debugging.
The reproducing example above consistently crashes on my local machine (SIGABRT, exit code 134):
But note that in the pybind11 tests, the underlying issue causes a different error (Python throwing a weird, seemingly unrelated exception). So something seems to be messed up in the interpreter internals, and the above example just triggers it.
vstinner commentedon Jan 11, 2021
Oh. In _PyUnicode_FromId(), I made the assumption that _PyRuntime is left unchanged when Py_Initialize()Py_Finalize() is called multiple times. But I was wrong, it is always reset to zero. So I wrote PR 24193 to explicitly save/restore _PyRuntime.unicode_ids.next_index value.
Using PR 24193, msg384761 example displays "Works" instead of failing with a Python fatal error.
YannickJadoul commentedon Jan 11, 2021
Wow, that was fast! Thanks!
I tried this out locally, and all pybind11's tests pass now. We can try again once there's a nightly build or new alpha :-)
vstinner commentedon Jan 12, 2021
New changeset 44bf57a by Victor Stinner in branch 'master':
bpo-42882: _PyRuntimeState_Init() leaves unicode next_index unchanged (GH-24193)
44bf57a
vstinner commentedon Jan 12, 2021
Thanks for your bug report, and thanks for testing alpha versions of Python! It's now fixed.
But I didn't feel comfortable without a regression test. So I wrote PR 24198 to add an unit test on _PyUnicode_FromId() with multiple Python initializations.
vstinner commentedon Jan 12, 2021
New changeset 11d13e8 by Victor Stinner in branch 'master':
bpo-42882: Add test_embed.test_unicode_id_init() (GH-24198)
11d13e8
vstinner commentedon Jan 12, 2021
Ok, now I can safely close the issue ;-)
vstinner commentedon Feb 4, 2021
New changeset 196d4de by Ken Jin in branch 'master':
bpo-42882: Fix MSVC warnings in pystate.c (GH-24440)
196d4de