Occasional RuntimeError from Condition.notify

[dougz]

BPO	22185
Nosy	@tim-one, @rhettinger, @pitrou, @vstinner
Files	fix.diff: move self._waiters modification inside lock

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-08-29.21:28:26.728>
created_at = <Date 2014-08-11.23:47:11.131>
labels = ['library', 'type-crash']
title = 'Occasional RuntimeError from Condition.notify'
updated_at = <Date 2014-08-29.21:28:26.726>
user = 'https://bugs.python.org/dougz'

bugs.python.org fields:

activity = <Date 2014-08-29.21:28:26.726>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2014-08-29.21:28:26.728>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2014-08-11.23:47:11.131>
creator = 'dougz'
dependencies = []
files = ['36351']
hgrepos = []
issue_num = 22185
keywords = ['patch']
message_count = 5.0
messages = ['225208', '225213', '226068', '226088', '226089']
nosy_count = 6.0
nosy_names = ['tim.peters', 'rhettinger', 'pitrou', 'vstinner', 'python-dev', 'dougz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue22185'
versions = ['Python 3.4', 'Python 3.5']

[dougz]

Condition.wait() modifies self._waiters without holding the lock (when a wait with timeout times out without the condition being notified).

If this happens to occur in between construction of the _islice and _deque objects in Condition.notify():

    def notify(self, n=1):
        [...]
        all_waiters = self._waiters
        waiters_to_notify = _deque(_islice(all_waiters, n))

then the result is a RuntimeError exception:

File "/usr/lib/python3.4/threading.py", line 358, in notify_all
  self.notify(len(self.\_waiters))
File "/usr/lib/python3.4/threading.py", line 341, in notify
  waiters_to_notify = \_deque(_islice(all_waiters, n))

RuntimeError: deque mutated during iteration

(I have a server which makes extensive use of conditions on which this happens about once a day.)

This patch fixes this bug by moving wait()'s modification of self._waiters to be inside the lock, as suggested by Antoine Pitrou here: http://bugs.python.org/issue17385#msg183875

[tim-one]

+1. I agree it's a bug, that the diagnosis is correct, and that the patch will fix it :-)

[dougz]

So, what happens now? What do I need to do to make progress on this?

[python-dev]

New changeset 4cce39cfe46c by Antoine Pitrou in branch '3.4':
Issue bpo-22185: Fix an occasional RuntimeError in threading.Condition.wait() caused by mutation of the waiters queue without holding the lock.
http://hg.python.org/cpython/rev/4cce39cfe46c

New changeset 78a38f8bd5d9 by Antoine Pitrou in branch 'default':
Issue bpo-22185: Fix an occasional RuntimeError in threading.Condition.wait() caused by mutation of the waiters queue without holding the lock.
http://hg.python.org/cpython/rev/78a38f8bd5d9

[pitrou]

It only needed someone to push your patch, which I just did.
Thank you very much Doug, your contribution is appreciated!