bpo-33530: Implement Happy Eyeballs in asyncio. #7097

twisteroidambassador · 2018-05-24T12:43:04Z

Added two keyword arguments, delay and interleave, to
BaseEventLoop.create_connection. Happy eyeballs is activated if
delay is specified.

https://bugs.python.org/issue33530

Added two keyword arguments, `delay` and `interleave`, to `BaseEventLoop.create_connection`. Happy eyeballs is activated if `delay` is specified.

twisteroidambassador · 2018-05-24T12:48:52Z

Happy Eyeballs functionality should be working. All old tests pass (except test_posix failing: there's an active issue about that bpo-33630). No new tests yet.

A bit of gymnastics was required to keep the old behavior w.r.t. handling of multiple exceptions.

1st1 · 2018-05-24T18:06:53Z

Lib/asyncio/helpers.py

@@ -0,0 +1,142 @@
+from contextlib import suppress


I'm usually against using module names like "helpers" or "utils", as it's impossible to know what's in them by just looking at the name. I propose to call this module "staggered.py".

1st1 · 2018-05-24T18:07:47Z

Lib/asyncio/helpers.py

@@ -0,0 +1,142 @@
+from contextlib import suppress


Please also import modules, not functions/classes directly. That's the style for asyncio code we try to maintain.

1st1 · 2018-05-24T18:10:27Z

Lib/asyncio/base_events.py

            local_addr=None, server_hostname=None,
-            ssl_handshake_timeout=None):
+            ssl_handshake_timeout=None,
+            delay=None, interleave=None):


Need to mention these in the docstring.

~~Is it a good idea to have one parameter instead of two? For example, interleave=-1 (default) is used to disable happy eyeballs. interleave>0 is used as a delay for happy eyeballs.~~

Oh, I misunderstood the meaning of interleave. Why shouldn't we always interleave addrs?

Why shouldn't we always interleave addrs?

Mostly because RFC8305 specifies "First Address Family Count" as a configurable value, and since I'm making it configurable, might as well provide the option of turning it off. Also, address family interleaving is a "SHOULD" in the RFC. If interleave is not specified, the default behavior is to interleave.

If we decide that the use case for configuring interleave is too small, we could remove it outright and just always interleave.

Need to mention these in the docstring.

The docstring of create_connection itself? The existing docstring doesn't explain all those old arguments, so I wondered whether it's appropriate to mention the new one.

(The larger Python docs should probably be updated as well?)

bpo-32374, bpo-33629: Use support.SuppressCrashReport() in test_bad_traverse() of MultiPhaseExtensionModuleTests to prevent leaking a core dump file.

Use uuid_enc_be() if available to encode UUID to bytes as big endian.

runtest_mp.py: call print() with flush=True.

…ythonGH-7111)

…ythonGH-6829)

Pickles of type variables and subscripted generics are now future-proof and compatible with older Python versions.

…pythonGH-7134) The failure may be due to the use oF ZFS, a case we already ignore for Solaris-based systems where ZFS is frequently used.

…ythonGH-7144)

…nGH-7145)

…l__ (python#7080)

…ythonGH-7149) Fixed bug where calling write_eof() on a _SelectorSocketTransport after it's already closed raises AttributeError.

…pythonGH-6836)

…pythonGH-7130) In this commit: * Support BufferedProtocol in set_protocol() and start_tls() * Fix proactor to cancel readers reliably * Update tests to be compatible with OpenSSL 1.1.1 * Clarify BufferedProtocol docs * Bump TLS tests timeouts to 60 seconds; eliminate possible race from start_serving * Rewrite test_start_tls_server_1

Using -w, when failing tests are re-run in verbose mode, display again the tests results at the end.

…ows are correctly regenerated. (pythonGH-7165)

…thonGH-7161)

Original patch by Dan O'Reilly.

pythonGH-7195)

…tions (pythonGH-7209)

…-7216) Currently, asyncio.wait_for(fut), upon reaching the timeout deadline, cancels the future and returns immediately. This is problematic for when *fut* is a Task, because it will be left running for an arbitrary amount of time. This behavior is iself surprising and may lead to related bugs such as the one described in bpo-33638: condition = asyncio.Condition() async with condition: await asyncio.wait_for(condition.wait(), timeout=0.5) Currently, instead of raising a TimeoutError, the above code will fail with `RuntimeError: cannot wait on un-acquired lock`, because `__aexit__` is reached _before_ `condition.wait()` finishes its cancellation and re-acquires the condition lock. To resolve this, make `wait_for` await for the task cancellation. The tradeoff here is that the `timeout` promise may be broken if the task decides to handle its cancellation in a slow way. This represents a behavior change and should probably not be back-patched to 3.6 and earlier.

…ython#7217) Unlike `asyncio.wait_for()`, `asyncio.wait()` does not cancel the passed futures when a timeout accurs.

…GH-7208)" (python#7232) This reverts commit 5d97b7b.

…" (pythonGH-7235) This reverts commit ad74d50. Turns out it's not a good fix -- Travis has just crashed on this test.

Added two keyword arguments, `delay` and `interleave`, to `BaseEventLoop.create_connection`. Happy eyeballs is activated if `delay` is specified.

twisteroidambassador · 2018-05-30T04:11:30Z

Uh, I did a rebase against upstream, and seems like it brought in commits from other people into this pull request. Should I close this PR and open a new one?

1st1 · 2018-05-30T04:13:24Z

Uh, I did a rebase against upstream, and seems like it brought in commits from other people into this pull request.

NP!

Should I close this PR and open a new one?

That's one way to fix it. Alternatively, you can just force-push a fixed branch and remove people from reviews/notifications.

1st1 · 2018-05-30T04:14:48Z

Seems there's no way we can remove devs that were accidentally subscribed to this PR. I'm closing this one, please submit a new PR.

Implement Happy Eyeballs in asyncio.

a755bbb

Added two keyword arguments, `delay` and `interleave`, to `BaseEventLoop.create_connection`. Happy eyeballs is activated if `delay` is specified.

twisteroidambassador requested review from 1st1 and asvetlov as code owners May 24, 2018 12:43

the-knights-who-say-ni added the CLA signed label May 24, 2018

bedevere-bot added the awaiting review label May 24, 2018

1st1 reviewed May 24, 2018

View reviewed changes

vstinner and others added 21 commits May 24, 2018 22:19

bpo-33629: Prevent coredump in test_importlib (pythonGH-7090)

483000e

bpo-32374, bpo-33629: Use support.SuppressCrashReport() in test_bad_traverse() of MultiPhaseExtensionModuleTests to prevent leaking a core dump file.

bpo-32493: Fix uuid.uuid1() on FreeBSD. (pythonGH-7099)

17d8830

Use uuid_enc_be() if available to encode UUID to bytes as big endian.

Document typing.NoReturn (pythonGH-7107)

6e413f4

test.regrtest: flush stdout when display progress (python#7105)

4f0bc7f

runtest_mp.py: call print() with flush=True.

bpo-33644: Fix signatures of tp_finalize handlers in testing code. (p…

19de8b3

…ythonGH-7111)

bpo-32769: A new take on annotations/type hinting glossary entries (p…

6e33f81

…ythonGH-6829)

Fix typo spotted by Guido (pythonGH-7131)

2298c0e

bpo-33652: Improve pickle support in the typing module. (pythonGH-7123)

09f3221

Pickles of type variables and subscripted generics are now future-proof and compatible with older Python versions.

bpo-33655: Also ignore test_posix_fallocate failures on BSD platforms (…

09c4a7d

…pythonGH-7134) The failure may be due to the use oF ZFS, a case we already ignore for Solaris-based systems where ZFS is frequently used.

bpo-32380: Create functools.singledispatchmethod (python#6306)

c651275

bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (p…

97b523d

…ythonGH-7144)

bpo-32374: Ignore Python-level exceptions in test_bad_traverse (pytho…

08c5aca

…nGH-7145)

bpo-33623: Fix possible SIGSGV when asyncio.Future is created in __de…

35230d0

…l__ (python#7080)

bpo-31647: Fix write_eof() after close() for SelectorSocketTransport (p…

23f587e

…ythonGH-7149) Fixed bug where calling write_eof() on a _SelectorSocketTransport after it's already closed raises AttributeError.

bpo-33505: Optimize asyncio.ensure_future by reordering if conditions (…

e549c4b

…pythonGH-6836)

regrtest: repeat summary after re-run (pythonGH-7159)

c6c05d0

Using -w, when failing tests are re-run in verbose mode, display again the tests results at the end.

bpo-33614: Ensures module definition files for the stable ABI on Wind…

e97ba4c

…ows are correctly regenerated. (pythonGH-7165)

bpo-32458: Further tune start_tls tests (python#7166)

8267ea2

bpo-33672: Fix Task.__repr__ crash with Cython's bogus coroutines (py…

989b9e0

…thonGH-7161)

bro-33614: Override exit code in find_msbuild.bat (pythonGH-7169)

f9b364f

1st1 and others added 20 commits May 29, 2018 13:40

Attempt to fix test_stdin_broken_pipe on Travis (python#7210)

4fadf0c

bpo-30935: update get_event_loop docs (pythonGH-2731)

e55de2d

bpo-22087: Fix Policy.get_event_loop() to detect fork (pythonGH-7208)

5d97b7b

Original patch by Dan O'Reilly.

bpo-31639: Change ThreadedHTTPServer to ThreadingHTTPServer class name (

1cee216

pythonGH-7195)

bpo-32684: Fix gather to propagate cancel of itself with return_excep…

863b674

…tions (pythonGH-7209)

bpo-23859: Document that asyncio.wait() does not cancel its futures (p…

f9aeca2

…ython#7217) Unlike `asyncio.wait_for()`, `asyncio.wait()` does not cancel the passed futures when a timeout accurs.

Fix whatsnew entry about ThreadedHTTPServer. (pythonGH-7220)

a34e424

bpo-32684: Fix nits in tests (pythonGH-7225)

6f75bae

Fix typo introduced in pythonGH-7217 (python#7230)

dec947c

Revert "bpo-22087: Fix Policy.get_event_loop() to detect fork (python…

99279ad

…GH-7208)" (python#7232) This reverts commit 5d97b7b.

A better fix for asyncio test_stdin_broken_pipe (python#7221)

ad74d50

Revert "A better fix for asyncio test_stdin_broken_pipe (pythonGH-7221)…

bb9474f

…" (pythonGH-7235) This reverts commit ad74d50. Turns out it's not a good fix -- Travis has just crashed on this test.

Implement Happy Eyeballs in asyncio.

c5d3a92

Added two keyword arguments, `delay` and `interleave`, to `BaseEventLoop.create_connection`. Happy eyeballs is activated if `delay` is specified.

Use module import instead of individual imports.

38c7caa

Change TODO.

70ec96d

Rename helpers.py to staggered.py.

cef0a76

Add create_connection()'s new arguments in documentation.

73a4a5a

Add blurb.

b3a6e1c

Merge remote-tracking branch 'origin/happy-eyeballs' into happy-eyeballs

632166d

twisteroidambassador requested review from a team, gpshead, gvanrossum, ilevkivskyi, ncoghlan and rhettinger as code owners May 30, 2018 04:08

1st1 closed this May 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-33530: Implement Happy Eyeballs in asyncio. #7097

bpo-33530: Implement Happy Eyeballs in asyncio. #7097

Uh oh!

twisteroidambassador commented May 24, 2018 •

edited by bedevere-bot

Loading

Uh oh!

twisteroidambassador commented May 24, 2018

Uh oh!

1st1 May 24, 2018

Uh oh!

1st1 May 24, 2018

Uh oh!

1st1 May 24, 2018 •

edited

Loading

Uh oh!

twisteroidambassador May 25, 2018

Uh oh!

twisteroidambassador May 25, 2018

Uh oh!

twisteroidambassador commented May 30, 2018

Uh oh!

1st1 commented May 30, 2018

Uh oh!

1st1 commented May 30, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Uh oh!

bpo-33530: Implement Happy Eyeballs in asyncio. #7097

bpo-33530: Implement Happy Eyeballs in asyncio. #7097

Uh oh!

Conversation

twisteroidambassador commented May 24, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twisteroidambassador commented May 24, 2018

Uh oh!

1st1 May 24, 2018

Choose a reason for hiding this comment

Uh oh!

1st1 May 24, 2018

Choose a reason for hiding this comment

Uh oh!

1st1 May 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twisteroidambassador May 25, 2018

Choose a reason for hiding this comment

Uh oh!

twisteroidambassador May 25, 2018

Choose a reason for hiding this comment

Uh oh!

twisteroidambassador commented May 30, 2018

Uh oh!

1st1 commented May 30, 2018

Uh oh!

1st1 commented May 30, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

twisteroidambassador commented May 24, 2018 •

edited by bedevere-bot

Loading

1st1 May 24, 2018 •

edited

Loading