Merge StreamWriter and StreamReader into just asyncio.Stream

[asvetlov]

BPO	36889
Nosy	@asvetlov, @1st1, @koobs, @aixtools, @icgood, @miss-islington, @tirkarthi
PRs	bpo-36889: Merge asyncio streams #13251 bpo-36889: Make StreamServer.close() tests more robust #13790 bpo-36889: Document Stream and StreamServer. #14203 [3.8] bpo-36889: Document asyncio Stream and StreamServer (GH-14203) #14349 bpo-36889: Replace deprecation warning with RuntimeError #14397 [3.8] bpo-36889: Replace deprecation warning with RuntimeError (GH-14397) #14422 bpo-36889: Document Stream class and add docstrings #14488 [3.8] bpo-36889: Document Stream class and add docstrings (GH-14488) #16087 bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" #16482 [3.8] bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482) #16485
Files	koobs-freebsd-current-3.x-build-168.stdio.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-13.11:32:11.525>
created_at = <Date 2019-05-11.18:15:14.485>
labels = ['3.8', 'expert-asyncio']
title = 'Merge StreamWriter and StreamReader into just asyncio.Stream'
updated_at = <Date 2019-09-30.05:30:20.394>
user = 'https://github.com/asvetlov'

bugs.python.org fields:

activity = <Date 2019-09-30.05:30:20.394>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-13.11:32:11.525>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2019-05-11.18:15:14.485>
creator = 'asvetlov'
dependencies = []
files = ['48376']
hgrepos = []
issue_num = 36889
keywords = ['patch']
message_count = 20.0
messages = ['342212', '343678', '343860', '343877', '343902', '343903', '344019', '344055', '344057', '344643', '344676', '346425', '346431', '346432', '346435', '346498', '352284', '352289', '353535', '353538']
nosy_count = 7.0
nosy_names = ['asvetlov', 'yselivanov', 'koobs', 'Michael.Felt', 'icgood', 'miss-islington', 'xtreak']
pr_nums = ['13251', '13790', '14203', '14349', '14397', '14422', '14488', '16087', '16482', '16485']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36889'
versions = ['Python 3.8']

[asvetlov]

The separation was a bad idea, sorry. Both classes are coupled too much.

1. Writer should know about the reader to make drain() work
2. Reader has no .close() method.

The idea is:

1. Merge StreamReader and StreamWriter into just a Stream.
2. The Stream should support three modes: readonly, writeonly, readwrite.
3. Keep StreamReader/StreamWriter but deprecate these classes.
4. Replace all asyncio internals to use Stream class.

[miss-islington]

New changeset 23b4b69 by Miss Islington (bot) (Andrew Svetlov) in branch 'master':
bpo-36889: Merge asyncio streams (GH-13251)
23b4b69

[koobs]

Seeing a test_ayncio failure on koobs-freebsd-current:

Timeout (0:25:00)!
Thread 0x0000000800abb000 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 558 in select
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 1813 in _run_once
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 563 in run_forever
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 595 in run_until_complete
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_asyncio/test_streams.py", line 1531 in test_stream_server_abort

Some errors (warnings?) later in the re-run (which passes), may be related/indicative:

Future exception was never retrieved
future: <Future finished exception=BrokenPipeError()>
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/asyncio/subprocess.py", line 173, in _feed_stdin
    await self.stdin.drain()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/asyncio/streams.py", line 1415, in drain
    await self._protocol._drain_helper()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/asyncio/streams.py", line 594, in _drain_helper
    await waiter
BrokenPipeError

Full log attached

[asvetlov]

Thanks for letting me know.
Investigating.
BrokenPipeError is not related, I had a plan to fix such messages during the beta phase.

[tirkarthi]

test_stream_shutdown_hung_task_prevents_cancellation was added as part of this issue. Seems to be a random error : https://ci.appveyor.com/project/python/cpython/builds/24901585

======================================================================
ERROR: test_stream_shutdown_hung_task_prevents_cancellation (test.test_asyncio.test_streams.StreamTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\projects\cpython\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 64] The specified network name is no longer available
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_asyncio\test_streams.py", line 1605, in test_stream_shutdown_hung_task_prevents_cancellation
    self.loop.run_until_complete(test())
  File "C:\projects\cpython\lib\asyncio\base_events.py", line 608, in run_until_complete
    return future.result()
  File "C:\projects\cpython\lib\test\test_asyncio\test_streams.py", line 1601, in test
    await task
  File "C:\projects\cpython\lib\test\test_asyncio\test_streams.py", line 1586, in client
    self.assertEqual(b'', await stream.readline())
  File "C:\projects\cpython\lib\asyncio\streams.py", line 1545, in readline
    line = await self.readuntil(sep)
  File "C:\projects\cpython\lib\asyncio\streams.py", line 1638, in readuntil
    await self._wait_for_data('readuntil')
  File "C:\projects\cpython\lib\asyncio\streams.py", line 1521, in _wait_for_data
    await self._waiter
  File "C:\projects\cpython\lib\asyncio\proactor_events.py", line 279, in _loop_reading
    data = fut.result()
  File "C:\projects\cpython\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "C:\projects\cpython\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 64] The specified network name is no longer available

---

[asvetlov]

Wow!
Thanks for the report.
Internally it is pretty close to freebsd problem.
The test is unstable, not asyncio code itself.
The test uses future objects to synchronize client and server socket processing but looks like there are race conditions still.

Please let me work on it.
If the problem is very annoying I can temporarily disable these problematic tests while working on their improvements

[koobs]

@andrew That would be great, as having failing buildbots can hide new regressions that end up taking much longer to identify and fix as they are hidden, and is better for you than reverting implementation code

[aixtools]

FYI: since:
commit 23b4b69
Author: Andrew Svetlov <andrew.svetlov@gmail.com>
Date: Mon May 27 22:56:22 2019 +0300

bpo-36889: Merge asyncio streams (GH-13251)



https://bugs.python.org/issue36889

AIX bot 'hangs' during
test_sendfile (test.test_asyncio.test_streams.StreamTests)

ctrl-c gives this (in case useful)
Warning -- 'test_asyncio' left behind file '@test_7012552_tmp'
Warning -- asyncio.events._event_loop_policy was modified by test_asyncio
Before: None
After: <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x307f8aa8>

== Tests result: INTERRUPTED ==
Test suite interrupted by signal SIGINT.

1 test omitted:
test_asyncio

Total duration: 13 min 36 sec
Tests result: INTERRUPTED
/data/prj/python/git/python3-3.8/Lib/asyncio/base_events.py:646: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <coroutine object StreamTests.test_sendfile.<locals>.test at 0x30576ed0>
Traceback (most recent call last):
  File "/data/prj/python/git/python3-3.8/Lib/test/test_asyncio/test_streams.py", line 1643, in test
    await do_connect(*srv.sockets[0].getsockname())
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/streams.py", line 314, in __aexit__
    await self.close()
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/streams.py", line 292, in close
    await tasks.wait([stream.close() for stream in streams])
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/streams.py", line 292, in <listcomp>
    await tasks.wait([stream.close() for stream in streams])
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/streams.py", line 1382, in close
    self._transport.close()
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/selector_events.py", line 680, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/base_events.py", line 711, in call_soon
    self._check_closed()
  File "/data/prj/python/git/python3-3.8/Lib/asyncio/base_events.py", line 504, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
/data/prj/python/git/python3-3.8/Lib/asyncio/streams.py:352: ResourceWarning: unclosed stream server <StreamServer>
  _warn(f"unclosed stream server {self!r}",
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/data/prj/python/git/python3-3.8/Lib/asyncio/selector_events.py:684: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=7>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/data/prj/python/git/python3-3.8/Lib/asyncio/selector_events.py:684: ResourceWarning: unclosed transport <_SelectorSocketTransport closing fd=8>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

[aixtools]

hmm - i had just synced with master. must have just missed something since there is a strike-out through #57460. If so, please ignore. BBL.

[vstinner]

Can this issue be closed now?

[asvetlov]

Please keep it until new streams will be documented.