Crash in gen_send_ex(): _PyErr_GetTopmostException() returns freed memory

[vstinner]

BPO	33996
Nosy	@vstinner, @ned-deily, @markshannon, @serhiy-storchaka, @1st1, @hroncok
Files	dagpool_test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-03.15:36:31.627>
created_at = <Date 2018-06-29.12:35:51.278>
labels = ['interpreter-core', '3.7', '3.8']
title = 'Crash in gen_send_ex(): _PyErr_GetTopmostException() returns freed memory'
updated_at = <Date 2018-07-03.15:36:31.626>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-07-03.15:36:31.626>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-03.15:36:31.627>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-06-29.12:35:51.278>
creator = 'vstinner'
dependencies = []
files = ['47658']
hgrepos = []
issue_num = 33996
keywords = ['3.7regression']
message_count = 6.0
messages = ['320712', '320714', '320832', '320833', '320854', '320987']
nosy_count = 6.0
nosy_names = ['vstinner', 'ned.deily', 'Mark.Shannon', 'serhiy.storchaka', 'yselivanov', 'hroncok']
pr_nums = []
priority = None
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33996'
versions = ['Python 3.7', 'Python 3.8']

[vstinner]

Attached dagpool_test.py script triggers a crash on Python 3.7, but works well on Python 3.6. The script is a highly simplified code of tests/dagpool_tests.py of eventlet.

To reproduce the crash, you only need a single dependency: greenlet. Example using a venv:

python3.7 -m venv ENV
ENV/bin/python -m pip install greenlet

Then run the script:

    $ test_venv/bin/python dagpool_test.py 
    Segmentation fault (core dumped)

eventlet bug report:
eventlet/eventlet#475

I suspected that the bug is caused by the new exc_info attribute of PyThreadState: commit ae3087c.

$ gdb -args test_venv/bin/python -X faulthandler dagpool_test.py
(gdb) run

Program received signal SIGSEGV, Segmentation fault.
0x000000000056c9d2 in PyErr_SetObject (exception=<type at remote 0x9709a0>, value=0x0) at Python/errors.c:101
101 Py_INCREF(exc_value);
(gdb) where
#0 0x000000000056c9d2 in PyErr_SetObject (exception=<type at remote 0x9709a0>, value=0x0) at Python/errors.c:101
#1 0x000000000056cd4e in PyErr_SetNone (exception=<type at remote 0x9709a0>) at Python/errors.c:162
#2 0x000000000067cb0c in gen_send_ex (gen=0x7fffea651d78, arg=0x0, exc=0, closing=0) at Objects/genobject.c:241
#3 0x000000000067dd86 in gen_iternext (gen=0x7fffea651d78) at Objects/genobject.c:542
#4 0x00000000005461b1 in _PyEval_EvalFrameDefault (...)
...

(gdb) p tstate
$1 = (PyThreadState *) 0xa132a0

(gdb) p tstate->exc_info
$2 = (_PyErr_StackItem *) 0x7fffea651930

(gdb) p *tstate->exc_info
$3 = {
exc_type = 0x0,
exc_value = 0x0,
exc_traceback = 0x0,
previous_item = 0x7fffea651d20
}

(gdb) p *tstate->exc_info->previous_item
$4 = {
exc_type = <unknown at remote 0xdbdbdbdbdbdbdbdb>,
exc_value = <unknown at remote 0xdbdbdbdbdbdbdbdb>,
exc_traceback = <unknown at remote 0xdbdbdbdbdbdbdbdb>,
previous_item = 0xdbdbdbdbdbdbdbdb
}

[vstinner]

I suspected that the bug is caused by the new exc_info attribute of PyThreadState: commit ae3087c.

The change comes from bpo-25612.

[markshannon]

Strange.
I could only reproduce this intermittently with 3.7 build.
But with a debug build, I cannot reproduce this at all.
Rebuilding Python (optimised, but without PGO) I cannot reproduce at all now.

[markshannon]

This looks like a Greenlet bug to me.
Possibly python-greenlet/greenlet@780a12b is incomplete.

It is not clear to me why greenlets hold their own exception state, but I suspect that if they didn't then this issue might go away.

[vstinner]

I could only reproduce this intermittently with 3.7 build.

Oh, I forgot to mention that I have to run the test many times to reproduce the crash. Between 1 and 10 times.

I used the master branch compiled in debug mode to report this issue.

[vstinner]

It's not a bug in Python but in greenlet:

• greenlet bug: Eventlet crashes with Python 3.7 python-greenlet/greenlet#131
• my greenlet fix: Fix #131: Port to Py3.7: handle tstate->exc_info python-greenlet/greenlet#132
• Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1594248