asyncio docs overhaul

[1st1]

BPO	33649
Nosy	@gvanrossum, @warsaw, @akuchling, @jrosdahl, @abalkin, @ned-deily, @asvetlov, @MartinAltmayer, @1st1, @willingc, @ilevkivskyi, @JDLH, @miss-islington, @tirkarthi, @kbumsik
PRs	bpo-33649: First asyncio docs improvement pass #9142 bpo-33649: Cleanup asyncio/streams and asyncio/synchronization docs sections #9192 bpo-33649: Fix doc to reflect changes in 47cd10d (or Issue #23347) #9219 bpo-33649: Edit asyncio eventloop doc - second pass #9233 bpo-33649: edit async policy doc - second pass #9235 bpo-33649 : polish asyncio exceptions and platforms docs #9272 bpo-33649: Polish asyncio subprocess and sync docs #9285 bpo-33649 Polish asyncio docs on queues, protocols, and subproccesses #9306 bpo-33649: Refresh Tasks and Futures pages #9314 bpo-33649: Add high-level APIs cheat-sheet #9319 bpo-33649: Refresh asyncio docs landing page #9322 bpo-33649: Fix asyncio-dev #9324 bpo-33649: Clarify protocol_factory as a method parameter #9330 bpo-33649: Add low-level APIs index. #9364 bpo-33649: Add a hello world example to asyncio.rst #9374 bpo-33649: A copy-editing pass on asyncio documentation #9376 [3.7] bpo-33649: Backport asyncio docs from 'master' #9377 bpo-33649: Note that asyncio.run() calls shutdown_asyncgens() #9380 [3.7] bpo-33649: Note that asyncio.run() calls shutdown_asyncgens() (GH-9380) #9383 bpo-33649: Fix syntax error on factorial example for asyncio #9385 bpo-33649: Change "set_after" reference to say_after. #9384 [3.7] bpo-33649: Change "set_after" reference to say_after. (GH-9384) #9386 bpo-33649: Fix syntax error on factorial example for asyncio #9387 [3.7] bpo-33649: Fix syntax error on Asyncio example in doc (GH-9387) #9388 bpo-33649: Fix markup; add another note that asyncio.run is 3.7+ #9389 [3.7] bpo-33649: Fix markup; add another note that asyncio.run is 3.7+ (GH-9389) #9390 bpo-33649: Add a high-level section about Futures; few quick fixes #9403 [3.7] bpo-33649: Add a high-level section about Futures; few quick fixes (GH-9403) #9409 bpo-33649: More improvements #9439 [3.7] bpo-33649: More improvements (GH-9439) #9454 bpo-33649: Fix gather() docs; fix title; few other nits. #9475 [3.7] bpo-33649: Fix gather() docs; fix title; few other nits. (GH-9475) #9481 bpo-40499: Mention that asyncio.wait() needs a non-empty aws set #19900

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-17.23:38:31.526>
created_at = <Date 2018-05-25.19:48:42.515>
labels = ['3.8', 'type-feature', '3.7', 'docs', 'expert-asyncio']
title = 'asyncio docs overhaul'
updated_at = <Date 2020-05-04.21:56:08.239>
user = 'https://github.com/1st1'

bugs.python.org fields:

activity = <Date 2020-05-04.21:56:08.239>
actor = 'gvanrossum'
assignee = 'docs@python'
closed = True
closed_date = <Date 2018-09-17.23:38:31.526>
closer = 'yselivanov'
components = ['Documentation', 'asyncio']
creation = <Date 2018-05-25.19:48:42.515>
creator = 'yselivanov'
dependencies = []
files = []
hgrepos = []
issue_num = 33649
keywords = ['patch']
message_count = 37.0
messages = ['317709', '317966', '318149', '320929', '323114', '325029', '325030', '325102', '325156', '325160', '325216', '325316', '325365', '325396', '325406', '325408', '325415', '325505', '325568', '325592', '325596', '325598', '325599', '325601', '325603', '325604', '325609', '325612', '325618', '325620', '325686', '325687', '325901', '325905', '326031', '326033', '368090']
nosy_count = 16.0
nosy_names = ['gvanrossum', 'barry', 'akuchling', 'jrosdahl', 'belopolsky', 'ned.deily', 'asvetlov', 'MartinAltmayer', 'docs@python', 'yselivanov', 'willingc', 'levkivskyi', 'JDLH', 'miss-islington', 'xtreak', 'kbumsik']
pr_nums = ['9142', '9192', '9219', '9233', '9235', '9272', '9285', '9306', '9314', '9319', '9322', '9324', '9330', '9364', '9374', '9376', '9377', '9380', '9383', '9385', '9384', '9386', '9387', '9388', '9389', '9390', '9403', '9409', '9439', '9454', '9475', '9481', '19900']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33649'
versions = ['Python 3.7', 'Python 3.8']

[1st1]

An overhaul of asyncio documentation is long overdue. Here's the structure for it that I have in mind:

• Introduction
  (what is asyncio and async/await)

• A quick tutorial
  (show how to use asyncio.run() and basic functions like asyncio.sleep() and teach that asyncio programs are all about async/await and *not* about callbacks or event loops)

• High-level APIs
  (Tasks, Streams, Subprocesses, few other functions)

• Low-level APIs

  • Preface (talk a bit about everything: what's an event loop, what is a Future and a Transport)
  • Futures
  • Event loop APIs
  • Transports and Protocols (when to use and when not to use them)

• Tutorials

  • High-level networking server
  • HTTP application
  • Low-level protocol implementation using Transports
  • etc

[akuchling]

The plan for updating things looks good, and I think the ordering of
tasks is good -- rearranging the APIs is a pretty simple first step,
and it matches other chapters in the Library Reference, which list
more important modules first and more specialized obscure modules
later. Then we can work on writing the additional material.

Yury: do you want to do the re-organization into high- and low-level APIs? If you'd like me to do it, I'll need a more explicit listing of which APIs fall into which category. Also, do we want to create an explicit high-level and low-level sections and push everything else down a level, or just leave it implicit in the ordering?

[1st1]

Yury: do you want to do the re-organization into high- and low-level APIs? If you'd like me to do it, I'll need a more explicit listing of which APIs fall into which category.

Sure, I'll update this issue tomorrow or the day after with a more detailed plan. Or maybe I'll start working on the 1st PR and will just give you push privileges to my repo. Would be cool if we can push the first rewrite live as soon as possible.

Also, do we want to create an explicit high-level and low-level sections and push everything else down a level, or just leave it implicit in the ordering?

Yes, I'd go for 2 explicitly separate sections. I think it's important to focus average readers' attention at the high-level bits and pieces, leaving low-level APIs and details to framework/library authors.

And, again, thanks for help.

Adding Carol to the nosy list as she seems to be interested in this topic too.

[jdlh]

I'm a developer using Python in my application. I just spent the last couple of weeks learning about asyncio with the present documentation. I am very happy to see that work is underway for improved documentation.

I would be glad to take on writing tasks or to review drafts, if there is a way to plug myself into that work.

I like the general structure.

Some specific topics which would have helped me in my recent learning:

• As an application developer using an event loop as part of the application, how do I develop a Protocol specific to my app, and use with Streams? An important part of this is being clear what the interface is between Transport and Protocol. This interface resides in both classes; Protocol calls Transport methods, and Transport calls Protocol methods.

• As an application developer using an event loop as part of the application, how do I develop a Transport specific to my app, and use it with Streams? (In my case, I have two transports in mind: a file, containing data archived from a TCP network port, and an RS232 serial connection, carrying data that might otherwise be sent via a TCP network port. It seems to me that I should be able at run-time to select from any of those three Transports, and apply my app-specific Protocol when creating my connection.)

• As an application developer, how do I give a function which is not a coroutine to the event loop for scheduling and execution? (I think the answer is loop.call_soon(), but the docs don't say "to run your function call loop.call_soon(myfunc...)", they say "call_soon(): Arrange for a callback to be called as soon as possible". Not the same thing. And, is a "callback" different "an arbitrary function"? Not clear.)

• As an application developer, how do I make an syncio-based streaming server and streaming client respond cleanly to interrupts, e.g. control-C in a command-line app? (Existing docs hints at catching signals, but don't show how to combine those with exception handlers in the coroutine to shut down connections cleanly.)

• As a Transport developer, what interface must I provide between Transport and Protocol? What services does the EventLoop provide to help implement my Transport? As mentioned before, to me it looks like the Transport-Protocol interface resides in both classes; Protocol calls Transport methods, and Transport calls Protocol methods. The EventLoop services useful to Transports are not gathered in a discussion of Transports. There is only an enumeration of the EventLoop methods, regardless of purpose.

Editorial checks I suggest making to the revision:

Rewrite to describe the behaviour in the doc, instead of delegating to a PEP. No more "this class is almost like the thing in PEP xyz, except for these differences". Describe the asyncio class, and steal text from PEP xyz as needed to do that.

Every section about a thing must start by saying what that thing is and what it is used for. There are some big counterexamples in the current text.

e.g. "19.5.1.2. Calls" <https://docs.python.org/3/library/asyncio-eventloop.html#calls\> starts out with: "Most asyncio functions don’t accept keywords. If you want to pass keywords to your callback, use functools.partial()." That does not describe what "calls" are, nor what they are used for. And, in this case, a better title might be, "Getting the event loop to call regular functions".

e.g. "19.5.1.4. Futures" <https://docs.python.org/3/library/asyncio-eventloop.html#futures\> has no starting text. This is the first heading mentioning Futures when reading sequentially in the asyncio doc, so I interpreted it as a place to learn about Futures. It is not. This section would be better titles "Eventloop methods for Futures". The proposed "Low-level APIs" section might solve this problem by explaining the nature and purpose of Futures before mentioning the EventLoop factory for Futures objects.

Consider separating the explanation of the nature and purpose of a thing from the reference to the methods of the thing. The proposed structure talks about "High-Level APIs" and "Low-Level APIs", but not "Architecture" and "API Reference". I would suggest injecting an "Architecture" section which can give the nature and purpose of each of the public classes in Asyncio, without enumerating their methods. Then the sections labelled "High-Level APIs" and "Low-Level APIs" can be the API reference.

I hope these suggestions are helpful. I'll be monitoring this issue to see how I can help.

[1st1]

Status update: I'm working on a first rewrite. I expect to have something to review on a couple of weeks.

[1st1]

New changeset 7c7605f by Yury Selivanov in branch 'master':
bpo-33649: First asyncio docs improvement pass (GH-9142)
7c7605f

[1st1]

The first PR has been just landed.

The plan is to work in 'master' branch and when the docs are ready we backport them to 3.7 in one big PR.

So far the following files were updated/reviewed:

 [Doc/library/asyncio-eventloop.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-eventloop.rst)
 [Doc/library/asyncio-eventloops.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-eventloops.rst) -> [Doc/library/asyncio-policy.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-policy.rst)
 [Doc/library/asyncio-exceptions.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-exceptions.rst)
 [Doc/library/asyncio-platforms.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-platforms.rst) 
 [Doc/library/asyncio-policy.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-policy.rst)    
 [Doc/library/asyncio-protocol.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-protocol.rst)  
 [Doc/library/asyncio-queue.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-queue.rst)     
 [Doc/library/asyncio-stream.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-stream.rst)    
 [Doc/library/asyncio-subprocess.rst](https://github.com/python/cpython/blob/main/Doc/library/asyncio-subprocess.rst) 

Please feel free to submit PR to fix/improve the above (but not other files as I'm going to be making some heavy edit passes on them).

[1st1]

New changeset 8be876e by Yury Selivanov in branch 'master':
bpo-33649: Cleanup asyncio/streams and asyncio/synchronization docs (GH-9192)
8be876e

[1st1]

New changeset aca819f by Yury Selivanov (Bumsik Kim) in branch 'master':
bpo-33649: Fix doc to reflect changes in 47cd10d (or bpo-23347) (GH-9219)
aca819f