Add top level await statement support for doctest

[tirkarthi]

BPO	37006
Nosy	@njsmith, @asvetlov, @1st1, @Carreau, @tirkarthi, @isidentical, @jack1142, @pacorain

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-22.09:20:50.115>
labels = ['type-feature', '3.9', 'expert-asyncio']
title = 'Add top level await statement support for doctest'
updated_at = <Date 2021-01-16.04:16:50.063>
user = 'https://github.com/tirkarthi'

bugs.python.org fields:

activity = <Date 2021-01-16.04:16:50.063>
actor = 'pacorain'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2019-05-22.09:20:50.115>
creator = 'xtreak'
dependencies = []
files = []
hgrepos = []
issue_num = 37006
keywords = []
message_count = 7.0
messages = ['343160', '343178', '343183', '343199', '343232', '343234', '343238']
nosy_count = 8.0
nosy_names = ['njs', 'asvetlov', 'yselivanov', 'mbussonn', 'xtreak', 'BTaskaya', 'jack1142', 'pacorain']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37006'
versions = ['Python 3.9']

[tirkarthi]

Since bpo-34616 is merged that allows using compile flags to support top level await statements I think it would be good to add support for top level await in doctest. This would help in concise examples in docs where await statements need to be wrapped in async def wrapper functions currently. This can be done using a doctest flag like ALLOW_TOP_LEVEL_AWAIT so that places where top level await is needed it can be explicitly marked as such so that when users copy paste code they are aware that it requires top level await statement.

I have implemented a simple patch where ALLOW_TOP_LEVEL_AWAIT flag (not to be confused with ast module flag) is added to doctest and if the doctest line has the flag then the ast flag is added th compileflags and then await eval(code_object) is used and then the awaitabe is executed with asyncio.run. Synchronous code has usual exec(code_object). I don't see any doctest failures with this patch against our Doc folder and test_doctest.

Few downsides is that it requires ast import for the flag value which could be little heavy but inspect module is already imported and I think it's an okay tradeoff for doctest. I have used asyncio.run and I am not sure if there is an efficient way to run awaitables. Feedback welcome.

Patch : master...tirkarthi:asyncio-await-doctest

# await_flag_doctest.rst
>>> import asyncio
>>> await asyncio.sleep(1.0)  # doctest: +ALLOW_TOP_LEVEL_AWAIT

cpython git:(asyncio-await-doctest) time ./python.exe -m doctest await_flag_doctest.rst
./python.exe -m doctest await_flag_doctest.rst 0.31s user 0.02s system 24% cpu 1.343 total

# await_no_flag_doctest.rst that will fail
>>> import asyncio
>>> await asyncio.sleep(1.0)

cpython git:(asyncio-await-doctest)   time ./python.exe -m doctest await_no_flag_doctest.rst
**********************************************************************
File "await_no_flag_doctest.rst", line 2, in await_no_flag_doctest.rst
Failed example:
    await asyncio.sleep(1.0)
Exception raised:
    Traceback (most recent call last):
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1338, in __run
        code = compile(example.source, filename, "single",
      File "<doctest await_no_flag_doctest.rst[1]>", line 1
    SyntaxError: 'await' outside function
**********************************************************************
1 items had failures:
   1 of   2 in await_no_flag_doctest.rst
***Test Failed*** 1 failures.
./python.exe -m doctest await_no_flag_doctest.rst  0.35s user 0.03s system 94% cpu 0.393 total

[asvetlov]

Please keep in mind: not only asyncio can be used to execute async/await code.

For example, trio has completely different implementation but utilizes async functions as well.

Probably we need to customize it, maybe by inheriting AsyncioDocTestRunner from DocTestRunner base class.

Also, using asyncio.run() for every line is not a good idea.
asyncio.run() creates an new loop instance every time.
Thus, the following doesn't work:
>>> session = aiohttp.ClientSession()
>>> async with session.get(url) as resp:
...     text = await resp.text()

The session is bound to loop, but the loop is changed by the next line.
loop.run_until_complete() is not ideal too: it creates a new task every time.
In AsyncioTestCase (#13386) I'm avoiding this problem by keeping a long-running task for saving the execution context between setup/teardown and test itself.

[tirkarthi]

I tried using AsyncioDocTestRunner that inherits from DocTestRunner and most of the current DocTestRunner is synchronous and the execution happens in __run that seems to cause problem due to name mangling inheriting and changing it. Also python -m doctest by default uses testmod/testfile that use DocTestRunner so I thought to change DocTestRunner would be simpler and existing code can use added doctest flag without changing runner.

To be little more clear by each line I meant each example. So in below "async with session.get(url) as resp:\n text = await resp.text()" counts as a single example whose code object is evaluated in asyncio.run which I said as per line by mistake.

>>> async with session.get(url) as resp:
...     text = await resp.text()

Your concerns are reasonable about asyncio.run per example seem to be over kill and might not work in few cases. I also didn't think about trio. I will look into those. Thanks for the pointers.

[Carreau]

The other thing to think about is ensure_future and create_task, they may not move forward until a foreground task is running.

You can keep a loop running between lines or code-chunks, but then doctest cannot contain asyncio.run().

I'm leaning toward conservatisme, and make things better but not perfect for 3.8; potentially improving, in minor release of 3.9

[njsmith]

As far as things like run/run_until_complete/etc go, I think doctests should have the same semantics as async REPLs, whatever those end up being.

Given that we don't actually have mature async REPLs, that the core feature to enable them only landed a few days ago, and the 3.8 freeze is almost here, I think we should defer the doctest discussion for 3.9. Hopefully by then, things will have settled down more in async REPL land, and it'll be more obvious what doctest should do.

[asvetlov]

Agree with Nathaniel.
There is no need to rush now.

[Carreau]

As a reference, PR from Yuri for an asyncREPL python -m asyncio which I believe he want in 3.8:

#13472

I'm also likely to align IPython behavior on whatever core python decides.