threading.enumerate(): reentrant call during a GC collection hangs

[vstinner]

BPO	44422
Nosy	@vstinner, @miss-islington
PRs	bpo-44422: Fix threading.enumerate() reentrant call #26727 [3.10] bpo-44422: Fix threading.enumerate() reentrant call (GH-26727) #26737 [3.9] bpo-44422: Fix threading.enumerate() reentrant call (GH-26727) #26738 bpo-44422: threading.Thread reuses the _delete() method #26741
Files	rec_threading.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-06-16.09:44:01.520>
created_at = <Date 2021-06-14.21:26:28.622>
labels = ['library', '3.9', '3.10', '3.11']
title = 'threading.enumerate(): reentrant call during a GC collection hangs'
updated_at = <Date 2021-06-16.09:44:01.519>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-06-16.09:44:01.519>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-16.09:44:01.520>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-06-14.21:26:28.622>
creator = 'vstinner'
dependencies = []
files = ['50110']
hgrepos = []
issue_num = 44422
keywords = ['patch']
message_count = 8.0
messages = ['395851', '395852', '395880', '395881', '395883', '395884', '395885', '395914']
nosy_count = 2.0
nosy_names = ['vstinner', 'miss-islington']
pr_nums = ['26727', '26737', '26738', '26741']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44422'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[vstinner]

The threading.enumerate() code is simple:
---

# Active thread administration
_active_limbo_lock = _allocate_lock()
_active = {}    # maps thread id to Thread object
_limbo = {}

def enumerate():
    with _active_limbo_lock:
        return list(_active.values()) + list(_limbo.values())

---

values(), list() and list+list operations can call trigger a GC collection depending on the GC thresholds, created Python objects, etc.

During a GC collection, a Python object destructor (del) can again call threading.enumerate().

Problem: _active_limbo_lock is not re-entrant lock and so the second call hangs.

In practice, this issue was seen in OpenStack which uses eventlet, when a destructor uses the logging module. The logging module uses threading.current_thread(), but this function is monkey-patched by the eventlet module.

Extract of the eventlet function:
---

def current_thread():
    ...
    found = [th for th in __patched_enumerate() if th.ident == g_id]
    ...

---

https://github.com/eventlet/eventlet/blob/master/eventlet/green/threading.py

Attached PR makes _active_limbo_lock re-entrant to avoid this issue. I'm not sure if it's ok or not, I would appreciate to get a review and your feedback ;-)

Attached file triggers the threading.enumerate() hang on re-entrant call.

[vstinner]

There are different ways to fix this issue:

• (A) Rewrite threading.enumerate() in C with code which cannot trigger a GC collection
• (B) Disable temporarily the GC
• (C) Use a reentrant lock (PR 26727)

(A) The problem is that functions other than threading.enumerate()
also rely on this lock, like threading.active_count(). I would prefer to not have to rewrite "half" of threading.py in C. Using a RLock is less intrusive.

(B) This is a simple and reliable option. But gc.disable() is process-wide: it affects all Python threads, and so it might have surprising side effects. Some code might rely on the current exact GC behavior. I would prefer to not disable the GC temporarily.

[vstinner]

New changeset 243fd01 by Victor Stinner in branch 'main':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727)
243fd01

[miss-islington]

New changeset c3b776f by Miss Islington (bot) in branch '3.10':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727)
c3b776f

[vstinner]

New changeset 8fe57aa by Miss Islington (bot) in branch '3.9':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727) (GH-26738)
8fe57aa

[vstinner]

Ok, the deadlock on reentrant call to threading.enumerate() is now fixed in 3.9, 3.10 and main (future 3.11) branches. I close the issue. Thanks for the reviews, as usual :-)