SystemError with bare `raise` in threading or multiprocessing

[romuald]

BPO	27558
Nosy	@rhettinger, @pitrou, @vstinner, @bitdancer, @berkerpeksag, @ztane, @romuald, @applio, @zhangyangyu, @Vgr255
Files	example.py: Reproducible example issue27558.patch issue27558_v2.patch: add test in test_threading instead of test_raise issue27558_v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/applio'
closed_at = <Date 2016-08-18.16:24:42.283>
created_at = <Date 2016-07-18.06:26:43.558>
labels = ['interpreter-core', 'type-bug']
title = 'SystemError with bare `raise` in threading or multiprocessing'
updated_at = <Date 2016-08-19.07:20:26.970>
user = 'https://github.com/romuald'

bugs.python.org fields:

activity = <Date 2016-08-19.07:20:26.970>
actor = 'SilentGhost'
assignee = 'davin'
closed = True
closed_date = <Date 2016-08-18.16:24:42.283>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2016-07-18.06:26:43.558>
creator = 'Romuald'
dependencies = []
files = ['43772', '43776', '43971', '44137']
hgrepos = []
issue_num = 27558
keywords = ['patch']
message_count = 24.0
messages = ['270713', '270728', '270729', '270730', '270733', '270734', '270735', '270746', '270755', '270757', '270759', '270761', '270763', '271802', '271822', '272407', '272409', '272436', '272938', '272943', '273011', '273043', '273044', '273046']
nosy_count = 11.0
nosy_names = ['rhettinger', 'pitrou', 'vstinner', 'r.david.murray', 'python-dev', 'berker.peksag', 'ztane', 'Romuald', 'davin', 'xiang.zhang', 'abarry']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27558'
versions = ['Python 3.5', 'Python 3.6']

[romuald]

Raising without a previous exception inside a method called from multiprocessing.dummy.Pool.map will trigger a SystemError.

SystemError: PyEval_EvalFrameEx returned NULL without setting an error

Traceback (most recent call last):
  File "example.py", line 16, in <module>
    main()
  File "example.py", line 11, in main
    result = pool.map(test, [1])
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
SystemError: PyEval_EvalFrameEx returned NULL without setting an error

Reproducible example is attached

[ztane]

Reproducible on Python 3.6a4ish on Ubuntu. I believe this needs forking multiprocessing.

do_raise is called with 2 NULLs as arguments, it should raise

PyErr_SetString(PyExc_RuntimeError,
                        "No active exception to reraise");

What happens is that PyThreadState is initialized to all NULL pointers on the new thread on multiprocessing, however type is expected to point to Py_None when no exception has been raised:

        PyThreadState *tstate = PyThreadState_GET();
        PyObject *tb;
        type = tstate->exc_type;
        value = tstate->exc_value;
        tb = tstate->exc_traceback;
        if (type == Py_None) {
            PyErr_SetString(PyExc_RuntimeError,
                            "No active exception to reraise");
            return 0;
        }

I am not sure where the thread state should have been initialized though

[ztane]

OTOH, if you put sys.exc_info() in place of raise there, it correctly (None, None, None) there, because it does (sysmodule.c:sys_exc_info)

    tstate->exc_type != NULL ? tstate->exc_type : Py_None,

Easiest fix would be to make do_raise test for both NULL and None, but I'd really consider fixing the new thread initialization if possible.

[ztane]

more easily reproducible by

    import threading
    def foo():
        raise
    threading.Thread(target=foo).start()

[zhangyangyu]

It seems the thread state is initialized in thread_PyThread_start_new_thread. It calls _PyThreadState_Prealloc which initialize exc_type to NULL.

[zhangyangyu]

I am curious why individual raise doesn't get the same problem in single thread program.

[ztane]

I was thinking that perhaps an exception is always raised somewhere before? I tried skipping site, but it still works, so I am not too sure.

[zhangyangyu]

Easiest fix would be to make do_raise test for both NULL and None, but I'd really consider fixing the new thread initialization if possible.

I think the easiest fix is OK. The new thread initialization is consistent and should not be changed. The Py_None seems to be set in somewhere in ceval but I am not sure where. You can see at https://hg.python.org/cpython/file/tip/Python/ceval.c#l1200 both NULL and Py_None are checked.

[zhangyangyu]

Upload a patch.

[berkerpeksag]

I would prefer Antti's test case in msg270730 since the bug can be reproduced by using threading module (you can use assert_python_ok() as a helper.) Lib/test_threading.py is probably a better place for the test.

[zhangyangyu]

It's reasonable but I'd still like to hear others' voices since it looks more like to me it's the behaviour of raise itself rather than threading that crashes.

[bitdancer]

Since ceval already has such a check, I think we should fix it by adding the check as proposed. Investigating whether the codebase could be simplified by making sure that Py_None is always there after startup would be a separate issue, I think, possibly related to PEP-432.