Spurious warning emitted during fork() can deadlock a multi-threaded process

[rojer]

BPO	40442
Nosy	@vstinner, @bowiechen, @rojer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-29.20:46:13.936>
labels = ['3.7', 'type-bug', 'library']
title = 'Spurious warning emitted during fork() can deadlock a multi-threaded process'
updated_at = <Date 2020-04-30.17:30:13.696>
user = 'https://github.com/rojer'

bugs.python.org fields:

activity = <Date 2020-04-30.17:30:13.696>
actor = 'bowiechen'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-04-29.20:46:13.936>
creator = 'rojer'
dependencies = []
files = []
hgrepos = []
issue_num = 40442
keywords = []
message_count = 1.0
messages = ['367701']
nosy_count = 4.0
nosy_names = ['vstinner', 'ionelmc', 'bowiechen', 'rojer']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40442'
versions = ['Python 3.7']

[rojer]

I know, I know - forking a multi-threaded process is bad. But it happens.

This is related to https://bugs.python.org/issue40399 and https://bugs.python.org/issue6721 but is distinct because the problem is entirely self-inflicted.

What happens:

1. A multithreaded program forks using one of the functions, such as os.forkpty()
2. In the child process the Python interpreter, in its PyOS_AfterFork_Child function ([1]) tries to kill all the threads other than the one doing the forking.
3. Among the objects being destroyed may include file or socket objects that are now being destroyed too, without having been previosuly closed, which triggers a ResourceWarning in the finalizer [2], [3].
4. Default action for warnings is to write to sys.stderr
5. A mutex used in BufferedIO is held by some other (now deceased thread).
6. Deadlock in _enter_buffered_busy [4].

This is bad because there is absolutely no way to avoid it without disabling warnings.
Even if the program is super careful to not do anything after forking other than exec, it doesn't help because the resource warning and the resulting deadlock is triggered by activity of the interpreter: it is the interpreter that orphans and is forcibly destroying the files and sockets, not the program that lost track of them.

[1]

cpython/Modules/posixmodule.c

Line 451 in beba1a8

PyOS_AfterFork_Child(void)

[2]

cpython/Modules/_io/fileio.c

Line 95 in beba1a8

if (PyErr_ResourceWarning(source, 1, "unclosed file %R", source)) {

[3]

cpython/Modules/socketmodule.c

Line 4800 in beba1a8

if (PyErr_ResourceWarning((PyObject *)s, 1, "unclosed %R", s)) {

[4]

cpython/Modules/_io/bufferedio.c

Line 282 in beba1a8

st = PyThread_acquire_lock(self->lock, 1);