Configuration of windows event loop for libraries

[bdarnell]

BPO	37373
Nosy	@njsmith, @asvetlov, @ambv, @bdarnell, @1st1, @zooba, @minrk, @cmeyer, @mikeshardmind, @jack1142, @carltongibson
Files	selector_thread.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-06-22.18:33:25.362>
labels = ['type-feature', '3.8', 'expert-asyncio']
title = 'Configuration of windows event loop for libraries'
updated_at = <Date 2021-05-15.19:32:35.216>
user = 'https://github.com/bdarnell'

bugs.python.org fields:

activity = <Date 2021-05-15.19:32:35.216>
actor = 'Ben.Darnell'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2019-06-22.18:33:25.362>
creator = 'Ben.Darnell'
dependencies = []
files = ['48434']
hgrepos = []
issue_num = 37373
keywords = []
message_count = 24.0
messages = ['346289', '346293', '346295', '346299', '346302', '346311', '346315', '346347', '346348', '346360', '346369', '353823', '355891', '355893', '355906', '355991', '362483', '362587', '362643', '362658', '366792', '366795', '393368', '393726']
nosy_count = 12.0
nosy_names = ['njs', 'asvetlov', 'lukasz.langa', 'Ben.Darnell', 'yselivanov', 'steve.dower', 'minrk', 'Big Stone', 'cmeyer', 'mikeshardmind', 'jack1142', 'carltongibson']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37373'
versions = ['Python 3.8']

Linked PRs

• gh-81554: Add add_reader support to ProactorEventLoop #141834

[bdarnell]

On Windows there are two event loop implementions with different interfaces: The proactor event loop is missing the file descriptor family of methods (add_reader()), while the selector event loop has other limitations including missing support for pipes and subprocesses and generally lower scalability. (The default has changed from selector to proactor in Python 3.8).

If an application requires the selector event loop, it can set the global event loop policy when it starts up. But what if a library requires the selector event loop? It wouldn't be appropriate for a library to set the event loop policy. The best I can do is document that "This library requires an event loop which supports the add_reader() method; to use this library on Windows (directly or indirectly) you must install the WindowsSelectorEventLoopPolicy."

This places a burden on application developers that target Windows to examine all their transitive dependencies to see if any require selectorevent loops (and if any have conflicting requirements for proactor event loops, which are even less likely to be documented since this is now the default). Concretely, this is a concern for Tornado (which requires add_reader()) and applications in the scientific python community (including Jupyter) which depend on it.

I know it's probably too late to do anything about this for 3.8, but it would be great if there were some better process to negotiate the right event loop. Some ideas (none of which are very satisfying):

• A declarative marker in setup.cfg or similar indicating the required feature set for an event loop
• A dummy package that could be depended on to indicate the same thing (and maybe this package could even use .pth hacks to change the default simply by being installed, but that seems like a hack too far)
• Some sort of runtime registry: at import time, Tornado could call asyncio.add_required_features(asyncio.ADD_READER) and this would influence the default policy
• Flags to get_event_loop() indicating the required features (passed through to the policy).

[1st1]

Victor, Andrew,

I'm not an expert in IOCP, but is it possible to implement add_reader/writer in ProactorEventLoop? If there's no native API for that, I guess we can spawn a thread with a 'select()' call to emulate this API?

Lukasz,

Another question: if we fix this, would you allow this to go in beta2/3? Strictly speaking it's going to be a new functionality.

[BigStone]

Windows users would certainly prefer to have an emulated non-performant emulation of the old API, rather than a breakage of their beloved Jupyter/Tornado ecosystem.

[asvetlov]

Is it possible to implement add_reader/writer in ProactorEventLoop?

Sorry, no. There is no way IIUC.

Regarding a request for selecting event loop by a library.

1. You can call asyncio.set_event_loop_policy() already.
2. From my understanding, there is no issue for Tornado itself. If Jupiter Notebook needs Tornado, Tornado needs selector event loop on Windows -- Jupiter can install the proper loop.

[bdarnell]

From my understanding, there is no issue for Tornado itself. If Jupiter Notebook needs Tornado, Tornado needs selector event loop on Windows -- Jupiter can install the proper loop.

Right. I'm just advocating for something that would make the transition smoother than each project independently stumbling across this issue and adding their own patch (and then fielding support issues from users who have ended up with a combination of versions that doesn't quite work). This of course depends on how many affected projects there are; I know Jupyter is the biggest but they're not the only one.

There's no more direct way, but a thread that does select() should work. In fact, ProactorEventLoop could simply spin up a SelectorEventLoop in another thread when one of the FD methods is called, and proxy the callbacks back and forth.

    def add_reader(self, fd, callback, *args):
        if not self.selector_started:
            self.start_selector()
        self.selector.call_soon(self.selector.add_reader, fd, lambda: self.call_soon(callback, *args))

    def start_selector(self):
        self.selector = SelectorEventLoop()
        def target():
            asyncio.set_event_loop(self.selector)
            self.selector.run_forever()
        thread = threading.Thread(target=target)
        thread.start()
        # Clean shutdown is left as an exercise for the reader.

Unifying the two interfaces like this would be preferable to adding more complexity for configuration IMHO.

[1st1]

Andrew,

Sorry, no. There is no way IIUC.

Well, in this case we should do that via a thread+select approach as I and Ben suggested.

I can write some PoC code; will you have some time to polish it and commit it? (I'm OOO and don't have a windows machine/VM at the moment).

[asvetlov]

I doubt if we should split a proactor loop into two objects in two threads.
I see no strong objections but have a feeling that we'll get other problems in this way. asyncio is just not designed for this mode.

The proposal looks like a very dirty hack and smells as it is.
There is no very strong reason to apply it.
Jupiter Notebook can fix the problem on its own side in two-three lines of clear code.

[bdarnell]

Yeah, it's definitely a hack. The argument for it, at best, is "practicality beats purity". The solution is two simple lines, but those lines need to be repeated in every project that depends on Tornado and cares about Windows, now or in the future. How many projects have to be affected to justify a hack like this? I'm not sure, but without this hack the add_reader methods, and by extension Tornado, will remain a compatibility trap.

[1st1]

+1 to what Ben said.

Andrew,

The proposal looks like a very dirty hack and smells as it is.

I wonder why are you feeling like that about this idea. I don't think this is a hack at all. There's no native API, so we're forced to use another approach.

There is no very strong reason to apply it.

I think there's such a reason. asyncio has a well defined API that people use and create programs dependent on it. And then people learn that the API isn't really supported on Windows with one loop, or when it is supported, other APIs aren't. They are then forced to either not support Windows or to rewrute their programs. The situation further complicates if they are using third-party code.

This is a very unfortunate and weird position we put our users into. So this indeed is a perfect example where "practicality beats purity" principle should be invoked.

[asvetlov]

I don't like to have not required parts like thread pool, time handlers, and the exception handler executed in the auxiliary thread.

What's about an alternative proposal: embed into ProactorEventLoop not entire SelectorEventLoop but selectors.SelectSelector only? We will have a little more code but the code is very clear and self-contained.

[vstinner]

Ben:

Concretely, this is a concern for Tornado (which requires add_reader()) and applications in the scientific python community (including Jupyter) which depend on it.

If you need add_reader/add_writer in Python 3.8, you can switch the default event loop to selector at the beginning of your application:

if sys.platform == 'win32':
  asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Yury:

If there's no native API for that, I guess we can spawn a thread with a 'select()' call to emulate this API?

There is no native API because IOCP design is to run an asynchronous read/write and then wait for its completion. Unix select() has the opposite design: check if a file descriptor is read for read/write.

Reimplementing add_reader/add_writer using a single select() call in a thread sounds like a good idea, but we have to make sure that it can be stopped whenever using a dedicated "self-pipe" (to awake the blocked select(), so loop.close() can stop this thread.

See attached proof-of-concept: selector_thread.py: run a selector in a separated thread which pass pack events to the loop using call_soon().

I would prefer to use a single selector to better scale with the number of FD.

Note: On Windows, select() only supports sockets.

Note: select.select() may be extended to support more than 512 sockets on Windows, see bpo-28708 :-)

Yury:

Another question: if we fix this, would you allow this to go in beta2/3? Strictly speaking it's going to be a new functionality.

It's a new feature, so it can wait for Python 3.9 :-)

I don't see any regression here, as soon as you can opt-in for SelectorEventLoop.

[njsmith]

It's also possible to implement 'select' on top of IOCP using undocumented private APIs. That's what libuv does (and how 'select' works internally). There's a bunch of information on this collected here: python-trio/trio#52