Fix possible SIGSGV when asyncio.Future is created in __del__

[1st1]

BPO	33623
Nosy	@vstinner, @ned-deily, @asvetlov, @serhiy-storchaka, @1st1, @miss-islington, @pbasista
PRs	bpo-33623: Fix possible SIGSGV when asyncio.Future is created in __del__ #7080 [3.7] bpo-33623: Fix possible SIGSGV when asyncio.Future is created in __del__ (GH-7080) #7151 [3.6] bpo-33623: Fix possible SIGSGV when asyncio.Future is created in __del__ (GH-7080) #7152
Files	uvloop-test.gdb-backtrace.txt: GDB backtrace showing the assertion failure while using uvloop

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-29.15:16:35.878>
created_at = <Date 2018-05-23.18:55:46.468>
labels = ['3.7', '3.8', 'release-blocker', 'type-crash', 'expert-asyncio']
title = 'Fix possible SIGSGV when asyncio.Future is created in __del__'
updated_at = <Date 2018-05-29.15:16:35.877>
user = 'https://github.com/1st1'

bugs.python.org fields:

activity = <Date 2018-05-29.15:16:35.877>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.15:16:35.878>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2018-05-23.18:55:46.468>
creator = 'yselivanov'
dependencies = []
files = ['47619']
hgrepos = []
issue_num = 33623
keywords = ['patch']
message_count = 12.0
messages = ['317437', '317440', '317441', '317444', '317602', '317658', '317858', '317863', '317864', '317868', '317930', '318026']
nosy_count = 7.0
nosy_names = ['vstinner', 'ned.deily', 'asvetlov', 'serhiy.storchaka', 'yselivanov', 'miss-islington', 'pbasista']
pr_nums = ['7080', '7151', '7152']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue33623'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[1st1]

Originally reported in MagicStack/uvloop#143

Future.__init__ shouldn't try to capture the current traceback if the interpreter is being finalized.

[serhiy-storchaka]

Is the problem because traceback_extract_stack is NULL?

But this will not fix the problem completely. For example calling repr() on a future can crash because of asyncio_future_repr_info_func == NULL. And many other operations use globals cleared in module_free().

[1st1]

Is the problem because traceback_extract_stack is NULL?

It's not NULL, it's just a broken reference at that point.

[serhiy-storchaka]

Then we should find what callable is NULL and fix the place where it is called. _PyObject_FastCallDict() should never be called with NULL.

[1st1]

Then we should find what callable is NULL and fix the place where it is called. _PyObject_FastCallDict() should never be called with NULL.

My understanding is that the interpreter is being shutdown and half of the objects are freed. We're still holding a reference to *something* in Python space and try calling it. The obvious fix for that is simply avoid capturing tracebacks if a Future object is created during finalization of the interpreter (it's pointless to capture it anyways at that point).

[serhiy-storchaka]

It is okay if the fact that half of the objects are freed leads to raising exceptions. But an assertion failure is sign of bugs in Python core or the _asyncio module. PR 7080 removes one way of exposing these bugs, but bugs itself still are here, and there may be other ways of triggering a crash.

Can this crash be reproduced without uvloop? What is the backtrace of the assertion failure?

[pbasista]

Can this crash be reproduced without uvloop?
I am not aware of a way to reproduce this without uvloop.

What is the backtrace of the assertion failure?
Without uvloop, the following exception is raised:

Exception ignored in: <bound method LoopWrapper.__del__ of <__main__.LoopWrapper object at 0x7fe14ec569b0>>
Traceback (most recent call last):
  File "uvloop_test.py", line 13, in __del__
  File "/usr/lib/python3.6/asyncio/base_events.py", line 276, in create_future
AttributeError: 'NoneType' object has no attribute 'Future'

With uvloop and with Python compiled with --with-pydebug configure option, the failed assertion is mentioned in the original uvloop bug report. In particular, this assertion fails:

python3.6-dbg: ../Objects/abstract.c:2300: _PyObject_FastCallDict: Assertion `func != NULL' failed.

A full GDB backtrace is attached.

[1st1]

There's nothing specific here to uvloop except the fact that it was compiled with Cython, so module finalization/GC pattern is slightly different to that of vanilla asyncio.

Serhiy, I'm merging this PR since I'd like to avoid having segfaults in 3.7.0. Feel free to debug this further.

[1st1]

New changeset 35230d0 by Yury Selivanov in branch 'master':
bpo-33623: Fix possible SIGSGV when asyncio.Future is created in __del__ (bpo-7080)
35230d0

[miss-islington]

New changeset 51d0a2c by Miss Islington (bot) in branch '3.7':
bpo-33623: Fix possible SIGSGV when asyncio.Future is created in __del__ (GH-7080)
51d0a2c

[1st1]

Would be great to merge this in 3.7.0. The change is super safe to merge.