UDP sockets created by create_datagram_endpoint() allow by default multiple processes to bind the same port

[vaizki]

BPO	37228
Nosy	@pitrou, @larryhastings, @ned-deily, @njsmith, @asvetlov, @ambv, @1st1, @dacut, @tirkarthi, @aeros
PRs	bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR #17311 [3.8] bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR (GH-17311) #17529 [3.7] bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR (GH-17311) (GH-17529) #17570 [3.6] bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR (GH-17311). #17571 bpo-37228: Fix warnings in test_asyncio.test_base_events #17577 [3.8] bpo-37228: Fix warnings in test_asyncio.test_base_events (GH-17577) #17579 [3.7] bpo-37228: Fix warnings in test_asyncio.test_base_events (GH-17577) #17580 [3.6] bpo-37228: Fix warnings in test_asyncio.test_base_events (GH-17577) #17581 bpo-37228: Add whatsnew for removal of loop.create_datagram_endpoint()'s *reuse_address* parameter #17595 [3.8] bpo-37228: Add whatsnew for removal of loop.create_datagram_endpoint()'s *reuse_address* parameter #17630 [3.7] bpo-37228: Add whatsnew for removal of loop.create_datagram_endpoint()'s *reuse_address* parameter #17631 [3.6] bpo-37228: Add whatsnew for removal of loop.create_datagram_endpoint()'s *reuse_address* parameter #17632

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-22.00:02:37.943>
created_at = <Date 2019-06-11.09:36:54.350>
labels = ['type-security', '3.7', '3.8', '3.9', 'expert-asyncio']
title = 'UDP sockets created by create_datagram_endpoint() allow by default multiple processes to bind the same port'
updated_at = <Date 2021-05-22.18:53:33.543>
user = 'https://bugs.python.org/vaizki'

bugs.python.org fields:

activity = <Date 2021-05-22.18:53:33.543>
actor = 'aeros'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-22.00:02:37.943>
closer = 'ned.deily'
components = ['asyncio']
creation = <Date 2019-06-11.09:36:54.350>
creator = 'vaizki'
dependencies = []
files = []
hgrepos = []
issue_num = 37228
keywords = ['patch']
message_count = 64.0
messages = ['345209', '345210', '356854', '356855', '356856', '356876', '356897', '356909', '356910', '356927', '356956', '357011', '357018', '357039', '357040', '357042', '357049', '357066', '357068', '357101', '357103', '357108', '357109', '357115', '357123', '357124', '357126', '357127', '357128', '357131', '357143', '357145', '357154', '357177', '357217', '357295', '357335', '357377', '357985', '357990', '358101', '358104', '358105', '358162', '358244', '358245', '358246', '358247', '358248', '358249', '358252', '358254', '358288', '358289', '358291', '358292', '358293', '358294', '358372', '358531', '362636', '394171', '394185', '394186']
nosy_count = 11.0
nosy_names = ['pitrou', 'larry', 'ned.deily', 'njs', 'asvetlov', 'lukasz.langa', 'yselivanov', 'dacut', 'xtreak', 'vaizki', 'aeros']
pr_nums = ['17311', '17529', '17570', '17571', '17577', '17579', '17580', '17581', '17595', '17630', '17631', '17632']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue37228'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[vaizki]

When using loop.create_datagram_endpoint(), the default for reuse_address=True, which sets the SO_REUSEADDR sockopt for the UDP socket.

This is a dangerous and unreasonable default for UDP, because in Linux it allows multiple processes to create listening sockets for the same UDP port and the kernel will randomly give incoming packets to these processes.

I discovered this by accidentally starting two Python asyncio programs with the same UDP port and instead of getting an exception that the address is already in use, everything looked to be working except half my packets went to the wrong process.

The documentation also refers to behavior with TCP sockets:
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.create_datagram_endpoint

"reuse_address tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire. If not specified will automatically be set to True on Unix."

This is true for TCP but not UDP.

Either the documentation should reflect the current (dangerous) behavior or the behavior should be changed for UDP sockets.

Workaround is of course to create your own socket without SO_REUSEADDR and pass it to create_datagram_endpoint().

[asvetlov]

There is no chance to change the flag for 3.7.
But we can consider it for 3.8

Yuri, what do you think?

[vaizki]

Can we still consider this for 3.9? I still think this is an easy way to accidentally blow your foot off with something that will probably only show up in production.

[asvetlov]

I see two proposals:

1. Change the default for reuse_address flag
2. Document existing behavior as dangerous.

What is the suggestion? Pick one, please :)

[vaizki]

I definitely propose changing the default for UDP sockets. Having multiple processes binding to a the same port and load-balancing incoming UDP traffic intentionally is a rare scenario which should be supported but not the default.

For TCP the default is reasonable because everyone creating server sockets will usually want to set it.

[1st1]

I'd be -1 on changing the default of an existing method, at least without consulting with a wider audience. We can add a new method to the loop and deprecate create_datagram_endpoint.

I suggest to post to python-dev and discuss this before making any decisions.

[vaizki]

Sure, I fully appreciate implications of changing default behaviour and will post on python-dev.

[gvanrossum]

I agree that this is a bad default (and whoever wrote it, probably me, didn't know what this does for UDP). I think the right solution is to change the default, not to introduce a new method.

Maybe we can deprecate the default somehow? There currently is a line the computes the default based on platform:

            if reuse_address is None:
                reuse_address = os.name == 'posix' and sys.platform != 'cygwin'

We could change this to issue a deprecation warning stating that in the future the default will change to False everywhere. Then in the next release we could change the default to False and remove the warning.

[njsmith]

Ouch, that's nasty. It also has security implications. For example, suppose you have a multi-user computer, where one user is has a video call going, which transfers packets over UDP. Another user could check what port they're using, bind to the same port, and steal half the packets, letting them effectively listen in on the first user's phone call.

Well, hopefully most conferencing tools use end-to-end encryption. But the point is that we don't normally let unprivileged users steal network traffic intended for other users.

[dacut]

How much harm would there be in bringing the DeprecationWarning into the next patch of existing (3.6, 3.7, 3.8) releases? The security implications are significant enough that I'd want to be notified of it in my software ASAP.

Users can (and should!) squelch the warning by passing the setting explicitly.