[subinterpreters] Fix pending calls in subinterpreters

[vstinner]

BPO	40231
Nosy	@vstinner, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-08.21:33:25.205>
labels = ['expert-subinterpreters', '3.9']
title = '[subinterpreters] Fix pending calls in subinterpreters'
updated_at = <Date 2020-05-15.00:39:53.889>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-05-15.00:39:53.889>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2020-04-08.21:33:25.205>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40231
keywords = []
message_count = 2.0
messages = ['366010', '366014']
nosy_count = 2.0
nosy_names = ['vstinner', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40231'
versions = ['Python 3.9']

[vstinner]

Currently, _Py_ThreadCanHandlePendingCalls() only returns true if the current thread is the Python "main thread" (_PyRuntime.main_thread).

_PyRuntime.main_thread is initialized by _PyRuntime_Initialize().

The problem is that a subinterpreter can run a separated thread which may not be the "main thread".

As a consequence, a subinterpreter will not run schedulded pending calls, or it will run them later than it could.

I modified COMPUTE_EVAL_BREAKER() of ceval.c in bpo-40010: now if tstate->interp->ceval.pending.calls_to_do is true, tstate->interp->ceval.eval_breaker is only set to 1 if _Py_ThreadCanHandlePendingCalls() is true.

One option would be to allow any thread to run "pending calls".

Another option is to have one "main thread" per interpreter, rather than having a single "main thread" for all interpreters.

I made pending calls per-interpreter in bpo-39984.

In Python 3.7, main_thread variable came from _PyRutimeState.ceval.pending.main_thread. It was moved into _PyRuntimeState by this change:

commit 5be45a6
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date: Fri Mar 8 22:47:07 2019 -0700

bpo-33608: Minor cleanup related to pending calls. (gh-12247)

--

_Py_ThreadCanHandleSignals() doesn't have to change: it must only return true for the main thread of the main interpreter. Currently, it's implemented as:

static inline int
_Py_ThreadCanHandleSignals(PyThreadState *tstate)
{
    return (_Py_IsMainThread() && _Py_IsMainInterpreter(tstate));
}

[vstinner]

I modified _PyEval_AddPendingCall() to accept interp rather than tstate to fix bpo-40082:

New changeset b54a99d by Victor Stinner in branch 'master':
bpo-40082: trip_signal() uses the main interpreter (GH-19441)
b54a99d

[vstinner]

I don't have the bandwidth to work on this issue, so I just close it.