threading module can deadlock after fork

[mikemccand]

BPO	874900
Nosy	@gpshead, @jcea, @amauryfa, @pitrou, @benjaminp
Files	fork_threading.py: demo script fork-and-threading5-gps01.patch: fixes fork-and-thread4, pitrou's fork_threading.py works with this patch. threading-874900-improvement-gps01.diff: is this an improvement? is it needed? forkthread.patch forkthread2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-09-06.23:04:57.802>
created_at = <Date 2004-01-11.14:28:06.000>
labels = ['interpreter-core', 'type-bug', 'release-blocker']
title = 'threading module can deadlock after fork'
updated_at = <Date 2011-07-11.06:04:45.798>
user = 'https://bugs.python.org/mikemccand'

bugs.python.org fields:

activity = <Date 2011-07-11.06:04:45.798>
actor = 'nirai'
assignee = 'none'
closed = True
closed_date = <Date 2008-09-06.23:04:57.802>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2004-01-11.14:28:06.000>
creator = 'mikemccand'
dependencies = []
files = ['9288', '10889', '10932', '11378', '11379']
hgrepos = []
issue_num = 874900
keywords = ['patch', 'needs review']
message_count = 54.0
messages = ['60453', '60454', '60455', '61693', '61695', '69423', '69437', '69466', '69467', '69468', '69470', '69476', '69477', '69478', '69479', '69492', '69512', '69549', '69550', '69569', '69601', '69603', '69607', '69696', '69702', '69784', '69793', '69795', '69825', '69867', '69868', '69878', '69888', '69889', '69893', '69894', '69928', '70174', '71297', '71298', '71301', '71315', '71818', '71824', '72546', '72549', '72550', '72551', '72552', '72591', '72716', '72717', '72721', '72727']
nosy_count = 10.0
nosy_names = ['gregory.p.smith', 'jcea', 'mikemccand', 'tzot', 'amaury.forgeotdarc', 'Rhamphoryncus', 'pitrou', 'benjamin.peterson', 'jnoller', 'bobbyi']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue874900'
versions = ['Python 3.0']

[mikemccand]

We have a Python HTTP server that, in the parent
process, uses os.fork to spawn new children, but at the
same time the parent could be spawning new threads (in
threads other than the main thread -- only the main
thread does forking).

Anwyay, it very rarely causes deadlock in a spawned
child when that child tries to start a new thread.

I believe I've tracked this down to the
_active_limbo_lock in the threading module.
Specifically, if this lock is held by parent (because
another thread is spawning a thread), just as os.fork
happens, the child inherits the held lock and will then
block trying to do any threading.* operations.

Just like the global interp. lock is overwritten in the
child after fork, I think something similar should
happen to threading._active_limbo_lock? (And more
generally the user needs to be aware of locks passing
through fork; but I think at least threading.py should
"do the right thing").

This thread looks quite relevant:

groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=38E6F2BA.E66CAC90%40ensim.com&rnum=5&prev=/groups%3Fq%3Dpython%2Bfork%2Bthreading%2Bmodule%2B%2Block%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26sa%3DN%26scoring%3Dd

[tzot]

Logged In: YES
user_id=539787

See some more typical info about mixing forks and threads:
http://mail.python.org/pipermail/python-list/2001-September/066048.html

This seems not to be Python-related, but platform-related.

[mikemccand]

Logged In: YES
user_id=323786

True -- there are many platform specific issues on the
interaction of forking and threading.

However the _active_limbo_lock is entirely a Python issue (I
think it can deadlock on any platform, unless the platform
releases locks in child process after fork).

After forking, python already resets the GIL, and it should
also reset any other locks that have global impact like
_active_limbo_lock.

[pitrou]

It's not only the _active_limbo_lock. All global structures of the
threading module should be reinitialized (including the _MainThread
instance); for that purpose, reload() can be used. I attach an example
which exercises this problem. Normally the script should hang in the
last step (using os.fork() and launching threads while some other
threads call threading.enumerate() in a loop).

The safe_fork() function in the example is a replacement for os.fork()
which tries to avoid any deadlock in the threading module. If it's
deemed useful and bug-free, it could perhaps be added somewhere in the
stdlib (in the threading module itself perhaps).

[pitrou]

(or perhaps we should provide an API to hook into PyOS_AfterFork)

[amauryfa]

Attached patch releases the _active_limbo_lock after a fork(). It is not
a complete solution, since existing Thread objects don't correspond to
anything, but it corrects a problem in test_multiprocessing.

[amauryfa]

A slightly better patch, with tests.

[jnoller]

Amaury - your latest patch doesn't seem to apply cleanly to trunk, it's
choking on the Python/ceval.c changes

[amauryfa]

Here is a third patch with latest trunk.
Did you apply it to a clean checkout?

[jnoller]

I had to flip on ignore whitespace to patch:
patch -p0 -l <~/Desktop/fork-and-thread3.patch

Worked. Unfortunately, test_threading is locking up during a make test.
Here's the verbose regrtest.py output:

woot:python-trunk jesse$ ./python.exe Lib/test/regrtest.py -v
test_threading
...snip
<thread 9> done
<thread 9> is finished. 0 tasks are running
all tasks done
ok
test_join_in_forked_from_thread
(test.test_threading.ThreadJoinOnShutdown) ...

At this point it hangs (OS/X 10.5 latest)

[jnoller]

Amaury, it looks like it's hanging in subprocess:

/Users/jesse/open_source/subversion/python-
trunk/Lib/test/test_threading.py(338)_run_and_join()
-> p = subprocess.Popen([sys.executable, "-c", script],
stdout=subprocess.PIPE)
(Pdb) step
...snip...

(Pdb) step
--Call--

/Users/jesse/open_source/subversion/python-
trunk/Lib/threading.py(788)current_thread()
-> def current_thread():
(Pdb) step
/Users/jesse/open_source/subversion/python-
trunk/Lib/threading.py(789)current_thread()
-> try:
(Pdb) step
/Users/jesse/open_source/subversion/python-
trunk/Lib/threading.py(790)current_thread()
-> return _active[_get_ident()]
(Pdb) step
/Users/jesse/open_source/subversion/python-
trunk/Lib/subprocess.py(1097)_execute_child()
-> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
(Pdb) step
... lockup