Using _multibytecodec module on Windows, test_threading/embed get failure

[neonene]

BPO	42846
Nosy	@pfmoore, @vstinner, @tjguk, @encukou, @zware, @zooba, @corona10, @shihai1991, @neonene, @erlend-aasland
PRs	bpo-42846: Convert CJK codec extensions to multiphase init #24157
Files	bug.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-07.23:18:25.235>
created_at = <Date 2021-01-06.22:38:21.954>
labels = ['extension-modules', 'type-bug', '3.10', 'expert-subinterpreters', 'tests', 'OS-windows']
title = 'Using _multibytecodec module on Windows, test_threading/embed get failure'
updated_at = <Date 2021-01-08.09:30:21.436>
user = 'https://github.com/neonene'

bugs.python.org fields:

activity = <Date 2021-01-08.09:30:21.436>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-07.23:18:25.235>
closer = 'vstinner'
components = ['Extension Modules', 'Tests', 'Windows', 'Subinterpreters']
creation = <Date 2021-01-06.22:38:21.954>
creator = 'neonene'
dependencies = []
files = ['49727']
hgrepos = []
issue_num = 42846
keywords = ['patch']
message_count = 12.0
messages = ['384541', '384607', '384610', '384611', '384613', '384616', '384618', '384619', '384620', '384621', '384622', '384644']
nosy_count = 10.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'petr.viktorin', 'zach.ware', 'steve.dower', 'corona10', 'shihai1991', 'neonene', 'erlendaasland']
pr_nums = ['24157']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42846'
versions = ['Python 3.10']

[neonene]

After 0b858cd
(bpo-1635741: Convert _multibytecodec to multi-phase init),

On Windows x64/x86 with chinese/japanese/korean system-locale,
MultibyteCodec_Check() in multibytecodec.c returns false and
PyExc_TypeError follows. This affects some tests and PGO training.

1. python -m test --verbose test_threading

======================================================================
FAIL: test_daemon_threads_fatal_error (test.test_threading.SubinterpThreadi
ngTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\cpython-0b858\lib\test\test_threading.py", line 1124, in test_da
emon_threads_fatal_error
    self.assertIn("Fatal Python error: Py_EndInterpreter: "
AssertionError: 'Fatal Python error: Py_EndInterpreter: not the last thread
' not found in 'TypeError: codec is unexpected type\nFatal Python error: _P
yThreadState_Delete: tstate 00000000003FF980 is still current\nPython runti
me state: initialized\n\nThread 0x00000710 (most recent call first):\n<no P
ython frame>\n'

2. python -m test --verbose test_embed

======================================================================
FAIL: test_audit_subinterpreter (test.test_embed.AuditingTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\cpython-0b858\lib\test\test_embed.py", line 1433, in test_audit_
subinterpreter
    self.run_embedded_interpreter("test_audit_subinterpreter")
  File "C:\cpython-0b858\lib\test\test_embed.py", line 104, in run_embedded
_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: 3221225477 != 0 : bad returncode 3221225477, stderr is 'Typ
eError: codec is unexpected type\nFatal Python error: _PyThreadState_Delete
: tstate 000000000050CAF0 is still current\nPython runtime state: initializ
ed\n\nThread 0x000009d8 (most recent call first):\n<no Python frame>\n'

======================================================================
FAIL: test_subinterps_different_ids (test.test_embed.EmbeddingTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\cpython-0b858\lib\test\test_embed.py", line 169, in test_subinte
rps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
  File "C:\cpython-0b858\lib\test\test_embed.py", line 110, in run_repeated
_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subint
erpreters")
  File "C:\cpython-0b858\lib\test\test_embed.py", line 104, in run_embedded
_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: 3221225477 != 0 : bad returncode 3221225477, stderr is 'Typ
eError: codec is unexpected type\nFatal Python error: _PyThreadState_Delete
: tstate 000000000041C960 is still current\nPython runtime state: initializ
ed\n\nThread 0x00000a40 (most recent call first):\n<no Python frame>\n'

======================================================================
FAIL: test_subinterps_distinct_state (test.test_embed.EmbeddingTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\cpython-0b858\lib\test\test_embed.py", line 177, in test_subinte
rps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
  File "C:\cpython-0b858\lib\test\test_embed.py", line 110, in run_repeated
_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subint
erpreters")
  File "C:\cpython-0b858\lib\test\test_embed.py", line 104, in run_embedded
_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: 3221225477 != 0 : bad returncode 3221225477, stderr is 'Typ
eError: codec is unexpected type\nFatal Python error: _PyThreadState_Delete
: tstate 000000000047C960 is still current\nPython runtime state: initializ
ed\n\nThread 0x00000b34 (most recent call first):\n<no Python frame>\n'

======================================================================
FAIL: test_subinterps_main (test.test_embed.EmbeddingTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\cpython-0b858\lib\test\test_embed.py", line 163, in test_subinte
rps_main
    for run in self.run_repeated_init_and_subinterpreters():
  File "C:\cpython-0b858\lib\test\test_embed.py", line 110, in run_repeated
_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subint
erpreters")
  File "C:\cpython-0b858\lib\test\test_embed.py", line 104, in run_embedded
_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: 3221225477 != 0 : bad returncode 3221225477, stderr is 'Typ
eError: codec is unexpected type\nFatal Python error: _PyThreadState_Delete
: tstate 000000000032C960 is still current\nPython runtime state: initializ
ed\n\nThread 0x00000bf0 (most recent call first):\n<no Python frame>\n'

[erlend-aasland]

I'm unable to reproduce this on Windows 10 (amd64). What's your exact locale setting? Are you compiling with HEAD at 0b858cd?

[vstinner]

I can reproduce the issue on Windows configured in Japanese language: ANSI code page cp932.

I managed to reproduce the bug on Linux with attached bug.py