await anext() returns None when default is given

[pewscorner]

BPO	43751
Nosy	@jab, @asvetlov, @1st1, @pablogsal, @pewscorner, @sweeneyde
PRs	bpo-43751: Fix anext() bug where it erroneously returned None #25238

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-11.04:52:03.772>
created_at = <Date 2021-04-06.16:00:14.125>
labels = ['interpreter-core', 'type-bug', 'release-blocker', '3.10', 'expert-asyncio']
title = 'await anext() returns None when default is given'
updated_at = <Date 2021-04-11.12:00:51.777>
user = 'https://github.com/pewscorner'

bugs.python.org fields:

activity = <Date 2021-04-11.12:00:51.777>
actor = 'pewscorner'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-11.04:52:03.772>
closer = 'pablogsal'
components = ['Interpreter Core', 'asyncio']
creation = <Date 2021-04-06.16:00:14.125>
creator = 'pewscorner'
dependencies = []
files = []
hgrepos = []
issue_num = 43751
keywords = ['patch']
message_count = 7.0
messages = ['390349', '390362', '390393', '390438', '390492', '390766', '390776']
nosy_count = 6.0
nosy_names = ['jab', 'asvetlov', 'yselivanov', 'pablogsal', 'pewscorner', 'Dennis Sweeney']
pr_nums = ['25238']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43751'
versions = ['Python 3.10']

[pewscorner]

The new anext() builtin in Python 3.10.0a7 doesn't seem to work properly when a default value is provided as the second argument. Here's an example:

import asyncio

async def f():
    yield 'A'
    yield 'B'

async def main():
    g = f()
    print(await anext(g, 'Z'))  # Prints 'None' instead of 'A'!!!
    print(await anext(g, 'Z'))  # Prints 'None' instead of 'B'!!!
    print(await anext(g, 'Z'))  # Prints 'Z'
    g = f()
    print(await anext(g))       # Prints 'A'
    print(await anext(g))       # Prints 'B'
    print(await anext(g))       # Raises StopAsyncIteration

asyncio.run(main())

As indicated above, anext() works fine when no default is given (in the second half of main()), but produces None in every iteration when a default is given (in the first half of main()) except when the iterator is exhausted.

[sweeneyde]

I can open a PR this evening, but I think this is close to the issue: PyIter_Next() already silences StopIteration, so checking for it afterward fails.

diff --git a/Objects/iterobject.c b/Objects/iterobject.c
index f0c6b79917..95f4659dc9 100644
--- a/Objects/iterobject.c
+++ b/Objects/iterobject.c
@@ -316,7 +316,7 @@ anextawaitable_traverse(anextawaitableobject *obj, visitproc visit, void *arg)
 static PyObject *
 anextawaitable_iternext(anextawaitableobject *obj)
 {
-    PyObject *result = PyIter_Next(obj->wrapped);
+    PyObject *result = (*Py_TYPE(obj->wrapped)->tp_iternext)(obj->wrapped);
     if (result != NULL) {
         return result;
     }

[sweeneyde]

That change fixes that bug, but I think there may be another bug involving when a custom async iterator is passed rather than an async generator. This is at the limit of my knowledge, so any guidance would be appreciated. The test I wrote in the PR currently fails, due to some tp_iternext slot being NULL sometimes. Maybe different cases are needed for Coroutine/non-Coroutine?

But it definitely seems like the aiter()/anext() code needs more test coverage.

[pewscorner]

Regarding the custom async iterator, I don't know if this is the problem you're referring to, but the following code seems to terminate abruptly when running main2() (main1() is fine). This is without your changes, though.

import asyncio

class CustomAsyncIter:
    def __init__(self):
        self.iterator = iter(['A', 'B'])
    def __aiter__(self):
        return self
    async def __anext__(self):
        try:
            x = next(self.iterator)
        except StopIteration:
            raise StopAsyncIteration from None
        await asyncio.sleep(1)
        return x

async def main1():
    iter1 = CustomAsyncIter()
    print(await anext(iter1))       # Prints 'A'
    print(await anext(iter1))       # Prints 'B'
    print(await anext(iter1))       # Raises StopAsyncIteration

async def main2():
    iter1 = CustomAsyncIter()
    print('Before')                 # Prints 'Before'
    print(await anext(iter1, 'Z'))  # Silently terminates the script!!!
    print('After')                  # This never gets executed

asyncio.run(main2())

[sweeneyde]

Okay, the PR should fix those problems now.

I am still apprehensive about whether all of the corner cases are covered, so reviews are welcome, as are suggestions of more test cases.