GIL destruction can fail

[pitrou]

BPO	9901
Nosy	@amauryfa, @pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-09-20.22:53:51.804>
created_at = <Date 2010-09-19.23:57:40.339>
labels = ['interpreter-core', 'type-bug']
title = 'GIL destruction can fail'
updated_at = <Date 2010-09-20.22:53:51.803>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2010-09-20.22:53:51.803>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-20.22:53:51.804>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2010-09-19.23:57:40.339>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 9901
keywords = []
message_count = 5.0
messages = ['116893', '116913', '116925', '116926', '116968']
nosy_count = 2.0
nosy_names = ['amaury.forgeotdarc', 'pitrou']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9901'
versions = ['Python 3.2']

[pitrou]

test_finalize_with_trace (in test_threading) sometimes fails because of failing to destroy the GIL (in _PyEval_FiniThreads()). This can be reproduced quite reliably by launching several copies in parallel:

$ ./python -m test.regrtest -j10 -F test_threading
[...]
test test_threading failed -- Traceback (most recent call last):
  File "/home/antoine/py3k/__svn__/Lib/test/test_threading.py", line 334, in test_finalize_with_trace
    "Unexpected error: " + ascii(stderr))
AssertionError: Unexpected error: b'Fatal Python error: pthread_mutex_destroy(gil_mutex) failed\n'

What happens is that pthread_mutex_destroy() fails with EBUSY. According to the POSIX man page:

“[EBUSY]
The implementation has detected an attempt to destroy the object referenced by mutex while it is locked or referenced (for example, while being used in a pthread_cond_timedwait() or pthread_cond_wait()) by another thread.”

After a bit of tracing, it becomes clear that Py_Finalize() calls _PyEval_FiniThreads() while another thread is taking the GIL (take_gil()). Unfortunately, this is not a situation we can avoid, since we rely on process exit to kill lingering threads: arbitrary CPython code may still be running in parallel while we are finalizing interpreter structures.

Therefore, it is likely that _PyEval_FiniThreads() should avoid destroying the mutex at all. Indeed, if we destroy the mutex, it is possible that a lingering thread tries to retake the GIL after waking up from a system call (Py_END_ALLOW_THREADS), and fails because of another fatal error ("Fatal Python error: pthread_mutex_lock(gil_mutex) failed").

[amauryfa]

looks similar to bpo-1856

[pitrou]

It is similar, but the issue is slightly different. Fixing bpo-1856 with the proposed resolution (the "stay_forever" flag) won't solve this, because the GIL mutex will still refuse to be destroyed if other threads reference it at the same time.

[pitrou]

Moving the _PyEval_FiniThreads() call to Py_Initialize() solves the issue:

diff -r 9e49082da463 Python/pythonrun.c
--- a/Python/pythonrun.c        Mon Sep 20 12:46:56 2010 +0200
+++ b/Python/pythonrun.c        Mon Sep 20 13:41:47 2010 +0200
@@ -217,8 +217,15 @@ Py_InitializeEx(int install_sigs)
         Py_FatalError("Py_Initialize: can't make first thread");
     (void) PyThreadState_Swap(tstate);
 
-    /* auto-thread-state API, if available */
 #ifdef WITH_THREAD
+    /* We can't call _PyEval_FiniThreads() in Py_Finalize because
+       destroying the GIL might fail when it is being referenced from
+       another running thread (see issue python/cpython#54110).
+       Instead we destroy the previously created GIL here, which ensures
+       that we can call Py_Initialize / Py_Finalize multiple times. */
+    _PyEval_FiniThreads();
+
+    /* Auto-thread-state API */
     _PyGILState_Init(interp, tstate);
 #endif /* WITH_THREAD */
 
@@ -514,10 +521,6 @@ Py_Finalize(void)
 
     PyGrammar_RemoveAccelerators(&_PyParser_Grammar);
 
-#ifdef WITH_THREAD
-    _PyEval_FiniThreads();
-#endif
-
 #ifdef Py_TRACE_REFS
     /* Display addresses (& refcnts) of all objects still alive.
      * An address can be used to find the repr of the object, printed

[pitrou]

Committed in r84927.