Throw away more radioactive locks that could be held across a fork in threading.py

[rnk]

BPO	6643
Nosy	@warsaw, @gpshead, @rnk
Files	forkjoindeadlock.py: Failing fork/threads test case. forkdeadlock.diff: Patch to fix the deadlock thread-fork-join.diff: Updated patch thread-fork-join.diff: clear condition waiters also issue6643-release26_maint_gps01.diff test_thread.diff: Patch to fix AttributeError in test_thread

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2011-01-04.18:44:17.436>
created_at = <Date 2009-08-04.18:56:48.860>
labels = ['interpreter-core', 'library', 'release-blocker']
title = 'Throw away more radioactive locks that could be held across a fork in threading.py'
updated_at = <Date 2013-07-30.10:54:40.366>
user = 'https://github.com/rnk'

bugs.python.org fields:

activity = <Date 2013-07-30.10:54:40.366>
actor = 'automatthias'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2011-01-04.18:44:17.436>
closer = 'gregory.p.smith'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2009-08-04.18:56:48.860>
creator = 'rnk'
dependencies = []
files = ['14647', '14648', '17932', '17936', '20249', '20259']
hgrepos = []
issue_num = 6643
keywords = ['patch', 'needs review']
message_count = 14.0
messages = ['91265', '91273', '109914', '109933', '110071', '110092', '125236', '125240', '125270', '125273', '125338', '125346', '125350', '193923']
nosy_count = 8.0
nosy_names = ['barry', 'collinwinter', 'gregory.p.smith', 'Rhamphoryncus', 'jyasskin', 'nadeem.vawda', 'rnk', 'automatthias']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue6643'
versions = ['Python 2.6']

[rnk]

This bug is similar to the importlock deadlock, and it's really part of
a larger problem that you should release all locks before you fork.
However, we can fix this in the threading module directly by freeing and
resetting the locks on the main thread after a fork.

I've attached a test case that inserts calls to sleep at the right
places to make the following occur:

• Main thread spawns a worker thread.
• Main thread joins worker thread.
• To join, the main thread acquires the lock on the condition variable
  (worker.__block.acquire()).
  == switch to worker ==
• Worker thread forks.
  == switch to child process ==
• Worker thread, which is now the only thread in the process, returns.
• __bootstrap_inner calls self.__stop() to notify any other threads
  waiting for it that it returned.
• __stop() tries to acquire self.__block, which has been left in an
  acquired state, so the child process hangs here.
  == switch to worker in parent process ==
• Worker thread calls os.waitpid(), which hangs, since the child never
  returns.

So there's the deadlock.

I think I should be able to fix it just by resetting the condition
variable lock and any other locks hanging off the only thread left
standing after the fork.

[rnk]

Here's a patch for 3.2 which adds the fix and a test case. I also
verified that the problem exists in 3.1, 2.7, and 2.6 and backported the
patch to those versions, but someone should review this one before I
upload those.

[rnk]

Here's an updated patch for py3k (3.2). The test still fails without the fix, and passes with the fix.

Thinking more about this, I'll try summarizing the bug more coherently:

When the main thread joins the child threads, it acquires some locks. If a fork in a child thread occurs while those locks are held, they remain locked in the child process. My solution is to do here what we do elsewhere in CPython: abandon radioactive locks and allocate fresh ones.

[rnk]

I realized that in a later fix for unladen-swallow, we also cleared the condition variable waiters list, since it has radioactive synchronization primitives in it as well.

Here's an updated patch that simplifies the fix by just using __init__() to completely reinitialize the condition variables and adds a test.

This corresponds to unladen-swallow revisions r799 and r834.

[Rhamphoryncus]

I don't have any direct opinions on this, as it is just a bandaid. fork, as defined by POSIX, doesn't allow what we do with it, so we're reliant on great deal of OS and library implementation details. The only portable and robust solution would be to replace it with a unified fork-and-exec API that's implemented directly in C.

[rnk]

I completely agree, but the cat is out of the bag on this one. I don't see how we could get rid of fork until Py4K, and even then I'm sure there will be people who don't want to see it go, and I'd rather not spend my time arguing this point.

The only application of fork that doesn't use exec that I've heard of is pre-forked Python servers. But those don't seem like they would be very useful, since with refcounting the copy-on-write behavior doesn't get you very many wins.

The problem that this bandaid solves for me is that test_threading.py already tests thread+fork behaviors, and can fail non-deterministically.

This problem was exacerbated while I was working on making the compilation thread.

I don't think we can un-support fork and threads in the near future either, because subprocess.py uses fork, and libraries can use fork behind the user's back.

[gpshead]

fwiw a unified fork-and-exec API implemented in C is what I added in Modules/_posixsubprocess.c to at least avoid this issue as much as possible when using subprocess.

[gpshead]

patch looks good. committed in r87710 for 3.2. needs back porting to 3.1 and 2.7 and optionally 2.6.

[gpshead]

r87726 for release31-maint
r87727 for release27-maint - this required a bit more fiddling as _block and _started and _cond were __ private.