Can't read data from Transport after asyncio.SubprocessStreamProtocol closes

[SethMichaelLarson]

BPO	29704
Nosy	@1st1, @SethMichaelLarson
PRs	bpo-29704: Fix asyncio.SubprocessStreamProtocol closing #405 bpo-29704: Fix asyncio.SubprocessStreamProtocol closing  #415 bpo-29704: Fix asyncio.SubprocessStreamProtocol closing #416 [Do Not Merge] Sample of CPython life with blurb. #703 [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-03-03.23:12:44.377>
created_at = <Date 2017-03-02.22:24:23.132>
labels = ['type-bug', 'expert-asyncio']
title = "Can't read data from Transport after asyncio.SubprocessStreamProtocol closes"
updated_at = <Date 2017-03-31.16:36:23.221>
user = 'https://github.com/SethMichaelLarson'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:23.221>
actor = 'dstufft'
assignee = 'none'
closed = True
closed_date = <Date 2017-03-03.23:12:44.377>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2017-03-02.22:24:23.132>
creator = 'SethMichaelLarson'
dependencies = []
files = []
hgrepos = []
issue_num = 29704
keywords = []
message_count = 4.0
messages = ['288839', '290327', '290333', '290335']
nosy_count = 2.0
nosy_names = ['yselivanov', 'SethMichaelLarson']
pr_nums = ['405', '415', '416', '703', '552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29704'
versions = []

[SethMichaelLarson]

Copied from python/asyncio#484

"""
From https://bugs.python.org/issue23242#msg284930

The following script is used to reproduce the bug:

import asyncio

async def execute():
    process = await asyncio.create_subprocess_exec(
        "timeout", "0.1", "cat", "/dev/urandom", stdout=asyncio.subprocess.PIPE)

    while True:
        data = await process.stdout.read(65536)
        print('read %d bytes' % len(data))
        if data:
            await asyncio.sleep(0.3)
        else:
            break

asyncio.get_event_loop().run_until_complete(execute())

will produce following output and terminate with exception:

read 65536 bytes
read 65536 bytes
Traceback (most recent call last):
  File "read_subprocess.py", line 18, in <module>
    asyncio.get_event_loop().run_until_complete(execute())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "read_subprocess.py", line 9, in execute
    data = await process.stdout.read(65536)
  File "/usr/lib/python3.6/asyncio/streams.py", line 634, in read
    self._maybe_resume_transport()
  File "/usr/lib/python3.6/asyncio/streams.py", line 402, in _maybe_resume_transport
    self._transport.resume_reading()
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 401, in resume_reading
    self._loop._add_reader(self._fileno, self._read_ready)
AttributeError: 'NoneType' object has no attribute '_add_reader'

When the process exits https://github.com/python/asyncio/blob/master/asyncio/unix_events.py#L444 is called which sets this._loop = None
Next time read() is called on the pipe the above exception is thrown.
I have tried to fix this issue myself but would sometimes have read terminate too early and miss the last chunks of data.
"""

• BotoX

[1st1]

New changeset f7f024a by Yury Selivanov (Seth M. Larson) in branch '3.5':
bpo-29704: Fix asyncio.SubprocessStreamProtocol closing (#405)
f7f024a

[1st1]

New changeset 604faba by Yury Selivanov (Seth M. Larson) in branch '3.6':
bpo-29704: Fix asyncio.SubprocessStreamProtocol closing (#405)
604faba

[1st1]

New changeset 481cb70 by Yury Selivanov (Seth M. Larson) in branch 'master':
bpo-29704: Fix asyncio.SubprocessStreamProtocol closing (#405)
481cb70