Asyncio reuseport

[Lothiraldan]

BPO	23972
Nosy	@gvanrossum, @vstinner, @Lothiraldan, @1st1
Files	23972_cjl.patch: Patch, tests and docs updates for 23972 23972_cjl_v002.patch: Updated patch, tests, docs for 23972 23972_cjl_v003.patch: rebase patch, tests, docs for 23972 23972_cjl_v004.patch: Patch, tests, docs updates for 23972 23972_cjl_v005.patch 23972_cjl_v006.patch: refine tests to also work on Windows and Debian platforms

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2015-10-05.16:33:28.391>
created_at = <Date 2015-04-16.11:52:15.669>
labels = ['type-bug', 'expert-asyncio']
title = 'Asyncio reuseport'
updated_at = <Date 2015-10-06.15:25:58.881>
user = 'https://github.com/Lothiraldan'

bugs.python.org fields:

activity = <Date 2015-10-06.15:25:58.881>
actor = 'python-dev'
assignee = 'gvanrossum'
closed = True
closed_date = <Date 2015-10-05.16:33:28.391>
closer = 'gvanrossum'
components = ['asyncio']
creation = <Date 2015-04-16.11:52:15.669>
creator = 'Boris.FELD'
dependencies = []
files = ['39930', '40230', '40616', '40664', '40667', '40696']
hgrepos = []
issue_num = 23972
keywords = ['patch']
message_count = 16.0
messages = ['241213', '241220', '246691', '246768', '246779', '249000', '251242', '251837', '251884', '252189', '252223', '252338', '252339', '252373', '252379', '252396']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'vstinner', 'Boris.FELD', 'python-dev', 'yselivanov', 'ysionneau', 'chris laws', 'j1o1h1n']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23972'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[Lothiraldan]

I'm trying to create some UDP sockets for doing multicast communication. I have the working code in synchronous way and try to port it to asyncio.

One last issue is blocking for me, I'm on Mac OS X and for multicast UDP to work, the SO_REUSEPORT must be set on socket before bind. The problem is that I don't have access on socket, it's created inside asyncio method _create_connection_transport.

I've seen that SO_REUSEADDR is already set (https://github.com/gvanrossum/tulip-try3/blob/7b2d8abfce1d7ef18ef516f9b1b7032172630375/asyncio/base_events.py#L720), so maybe we could also set SO_REUSEPORT only on platforms where it's available. if hasattr(socket, 'SO_REUSEPORT') should works.

Or we could add an optional arguments with could be used to set some socket options, it could be more flexible that set SO_REUSEPORT.

I could provide a patch for the best solution selected.

[gvanrossum]

I think we left this unfinished. I think it would be best if we added a sock parameter (like for create_server()) but also added reuse_address and reuse_port parameters, treated similarly like the one on create_server(), and added a reuse_port parameter to create_server().

Or maybe there should only be one parameter that guides both; read http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t before you decide!

[chrislaws]

I encountered this issue too. I needed it resolved ASAP for my work so I created a loop patch that partially implements the suggestion solution by overriding the create_datagram_endpoint method. Perhaps this might be of some use to the eventual ticket resolver.

It can be found in the following gist: https://gist.github.com/claws/d4076b4b155695844d54

If this looks like it's progressing in the right direction I can try to implement it properly in base_events.py and submit it via a patch.

[chrislaws]

Attached is a patch that implements the suggested solution along with tests and associated doc updates. Hope this helps.

[vstinner]

A quick search told me that "Windows only knows the SO_REUSEADDR option, there is no SO_REUSEPORT". It should be at least documented that SO_REUSEADDR is not supported on Windows.

Maybe we should raise an exception on Windows if reuse_port=True? Ignoring the parameter is a different option, but it should be well documented that the option is ignored if socket.SO_REUSEPORT is not defined.

[chrislaws]

I have updated the patch to address comments raised by haypo.

An exception is now raised if reuse_port is explicitly used and the platform does not support SOREUSEPORT. The docs have also been updated to make it more explicit that this feature is not supported on Windows.

[vstinner]

Hum, the latest patch was not reviewed yet :-(

[chrislaws]

Rebase patch onto current master.

[gvanrossum]

I added a whole bunch of review comments. Please send a new patch!

[chrislaws]

Updates addressing review comments.

[gvanrossum]

I'm adding a rebased version of Chris's v4 patch to see if I can get the code review to trigger.

[python-dev]

New changeset 5e7e9b131904 by Guido van Rossum in branch '3.4':
Issue bpo-23972: updates to asyncio datagram API. By Chris Laws.
https://hg.python.org/cpython/rev/5e7e9b131904

New changeset ba956289fe66 by Guido van Rossum in branch '3.5':
Issue bpo-23972: updates to asyncio datagram API. By Chris Laws. (Merge 3.4->3.5.)
https://hg.python.org/cpython/rev/ba956289fe66

New changeset c0f1f882737c by Guido van Rossum in branch 'default':
Issue bpo-23972: updates to asyncio datagram API. By Chris Laws. (Merge 3.5->3.6.)
https://hg.python.org/cpython/rev/c0f1f882737c

[gvanrossum]

Thanks for the patch! It's live now. Closing. Watch the buildbots for me please!