asyncio with two interpreter instances

[mbadaire]

BPO	47219
Nosy	@asvetlov, @ericsnowcurrently, @1st1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-04.19:57:08.003>
labels = ['type-bug', 'expert-asyncio']
title = 'asyncio with two interpreter instances'
updated_at = <Date 2022-04-04.20:38:00.422>
user = 'https://bugs.python.org/mbadaire'

bugs.python.org fields:

activity = <Date 2022-04-04.20:38:00.422>
actor = 'JelleZijlstra'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2022-04-04.19:57:08.003>
creator = 'mbadaire'
dependencies = []
files = []
hgrepos = []
issue_num = 47219
keywords = []
message_count = 1.0
messages = ['416694']
nosy_count = 4.0
nosy_names = ['asvetlov', 'eric.snow', 'yselivanov', 'mbadaire']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47219'
versions = []

• PR: GH-91375: Port _asyncio static types to heap types and module state #99122

• PR: GH-91375: Purge asyncio variables from globals-to-fix list #101288

[mbadaire]

Hi,
I have an issue when using asyncio and two interpreter instances each launched and used in a seperated thread.
I am getting a asyncio loop for each thread .However asyncio is getting me the same loop because of this code in get_running_loop. Indeed when I have two interpreter, the ts_id would be the same for both my threads and therefore I will get the cached value of the first thread. cached_running_holder being static, it is the same value for all instances of interpreter.
Maybe we should check if we are in the same interpreter or same thread ,.. I am not sure how it could be fixed.

_asynciomodule.c:
get_running_loop(PyObject **loop)
{
    PyObject *rl;

    PyThreadState *ts = _PyThreadState_GET();
    uint64_t ts_id = PyThreadState_GetID(ts);
    if (ts_id == cached_running_holder_tsid && cached_running_holder != NULL) {

If it does not make sense, I have some sample code but it is not just 10 lines.

[pipiche38]

hello, is there anything we can do to get some attention ?
This is a blocking issue in our project which spawn several instances from a Python embedded framework

Thanks in advance

[ericsnowcurrently]

It looks like asyncio has not been updated to support use in multiple interpreters -- the module uses a whole bunch of global variables to store state. Until that is dealt with, asyncio cannot be used reliably with multiple interpreters. See PEP 687. @erlend-aasland

Also note that this is unlikely to be considered a bug, so no backports. Likewise it is unlikely to happen for 3.11 since the feature freeze is literally today. Thus the soonest you may see a fix is 3.12 (Fall 2023), unfortunately.

[erlend-aasland]

Yeah, as soon as PEP 687 land, we can start the process of implementing this. I believe I've got a WIP branch lying around already.

[pipiche38]

Thanks for the update, we will be patient. just a shame not to have back port, but understood the amount of work.
Now, we would be really please to test it when you want .

Potential testing platforms OS: raspian or fedora

[erlend-aasland]

I can push it to my fork next week, so you can play with it.

[kumaraditya303]

One workaround currently is to not use _asyncio speed up module but to use pure python version as that does not has this limitation.

[pipiche38]

One workaround currently is to not use _asyncio speed up module but to use pure python version as that does not has this limitation.

interesting. How to you prevent using _asyncio ?

[kumaraditya303]

Before importing asyncio add this line:

import sys
sys.modules["_asyncio"] = None