asyncio.create_task() documentation should mention user needs to keep reference to the task

@cmeyer

BPO	44665
Nosy	@cmeyer, @nanjekyejoannah, @vincentbernat, @alexhartl
PRs	bpo-44665: asyncio.create_task() documentation should mention user needs to keep reference to the task #29163

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-18.06:01:26.261>
labels = ['3.11', 'type-feature', '3.9', '3.10', 'docs']
title = 'asyncio.create_task() documentation should mention user needs to keep reference to the task'
updated_at = <Date 2022-01-06.08:53:46.602>
user = 'https://github.com/vincentbernat'

bugs.python.org fields:

activity = <Date 2022-01-06.08:53:46.602>
actor = 'alexhartl'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2021-07-18.06:01:26.261>
creator = 'bernat'
dependencies = []
files = []
hgrepos = []
issue_num = 44665
keywords = ['patch']
message_count = 5.0
messages = ['397741', '404776', '404782', '404827', '409821']
nosy_count = 5.0
nosy_names = ['docs@python', 'cmeyer', 'nanjekyejoannah', 'bernat', 'alexhartl']
pr_nums = ['29163']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44665'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

asyncio will only keep weak references to alive tasks (in _all_tasks). If a user does not keep a reference to a task and the task is not currently executing or sleeping, the user may get "Task was destroyed but it is pending!".

I would suggest adding the following paragraph to create_task() documentation:

Python only keeps weak references to the scheduled tasks. To avoid the task being destroyed by the garbage collector while still pending, a reference to it should be kept until the task is done.

And maybe an example in case a user wants something "fire and forget"?

running_tasks = set()
# [...]
task = asyncio.create_task(some_background_function())
running_tasks.add(task)
task.add_done_callback(lambda t: running_tasks.remove(t))

The same applies to ensure_future as it now uses create_task, so maybe a "See create_task()".

@bernat

@bernat and ncoghlan, please see if the wording I have used in the linked PR helps to clarify this.

Is there a way to reproduce this issue? I run the following code in Python 3.9 and it works as expected (prints "xyz" twice).

import asyncio
import gc

async def xyz():
    print("xyz")

event_loop = asyncio.get_event_loop()

event_loop.create_task(xyz())

t = event_loop.create_task(xyz())
del t

gc.collect()

event_loop.stop()
event_loop.run_forever()

Ensure we hold strong references to tasks

util.CallbackManager: keep strong references for running futures

Is there a way to reproduce this issue? I run the following code in Python 3.9 and it works as expected (prints "xyz" twice).

import asyncio
import gc

async def xyz():
    print("xyz")

event_loop = asyncio.get_event_loop()

event_loop.create_task(xyz())

t = event_loop.create_task(xyz())
del t

gc.collect()

event_loop.stop()
event_loop.run_forever()

Here is my answer for anyone wondering how to reproduce this issue (as their code works just fine without keeping a reference to the tasks). I tried to explain what I understood:

https://stackoverflow.com/a/76823668/13944524

util.CallbackManager: keep strong references for running futures

@vincentbernat

@vincentbernat just to clarify, since I'm not that familiar with the intricacies of Python's garbage collector.

Is this enough (in the context of this issue) to keep a reference of the task like this?

async def coro():
    task1 = asyncio.create_task(...)
    task2 = asyncio.create_task(...)
    task3 = asyncio.create_task(...)
    task4 = asyncio.create_task(...)

Or is it really needed to keep them in a collection?

If so, can the collection be declared inside the coroutine? Or does it need to be outside to be a "strong" reference?

async def coro():
    running_tasks = set()
    task1 = asyncio.create_task(...)
    running_tasks.add(task1)
    task2 = asyncio.create_task(...)
    running_tasks.add(task2)
    task3 = asyncio.create_task(...)
    running_tasks.add(task3)
    task4 = asyncio.create_task(...)
    running_tasks.add(task4)

vs.

running_tasks = set()

async def coro():
    task1 = asyncio.create_task(...)
    running_tasks.add(task1)
    task2 = asyncio.create_task(...)
    running_tasks.add(task2)
    task3 = asyncio.create_task(...)
    running_tasks.add(task3)
    task4 = asyncio.create_task(...)
    running_tasks.add(task4)

Just trying to keep the code as simple as possible.

Only the last one will work. The two other ones don't keep a reference on the tasks after the coro() function ends.

Only the last one will work. The two other ones don't keep a reference on the tasks after the coro() function ends.

But when the coro() function ends, all the tasks created inside coro() are done, so I don't need to keep the references anymore, right?

Maybe I should have clarified that I call the await inside the same coro() function.

So like:

async def coro():
    task1 = asyncio.create_task(...)
    task2 = asyncio.create_task(...)
    task3 = asyncio.create_task(...)
    task4 = asyncio.create_task(...)
    await task1
    await task2
    await task3
    await task4
    # all the tasks are done and coro() ends

In this case, would it work? Do I even need to keep the references task1, task2, etc or can I do directly away asyncio.create_task(...) without risking my tasks to be garbage collected before they are done?

async def coro():
    await asyncio.create_task(...)
    await asyncio.create_task(...)
    await asyncio.create_task(...)
    await asyncio.create_task(...)
    # all the tasks are done and coro() ends

Thanks for the help.

Editing this post to avoid creating more noise:

I don't think this issue should be a support forum.

Agree but the documentation was updated following this issue and the wording is confusing. As you can see in the previous comments and in mine, it raises a lot of questions. The doc suggests to keep references of the tasks in any case, while apparently is not always necessary (like in my example).

I appreciate the help anyway. Thanks.

I don't think this issue should be a support forum. But yes, both ways are fine (but they don't do the same thing).

[BugFix] Prevent the task of `_force_log` from being garbage collected

~~Is there any reason why the event loop should not keep strong reference to the task until the task is completed?~~
-- never mind, this is being discussed already here: #91887

util.CallbackManager: keep strong references for running futures

Navigation Menu

Repository navigation

asyncio.create_task() documentation should mention user needs to keep reference to the task #88831

22 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

asyncio.create_task() documentation should mention user needs to keep reference to the task #88831

Description

Activity

vincentbernat commented on Jul 18, 2021

nanjekyejoannah commented on Oct 22, 2021

cmeyer commented on Oct 22, 2021

22 remaining items

amirsoroush commented on Aug 5, 2023

ggregoire commented on Feb 28, 2024

vincentbernat commented on Feb 28, 2024

ggregoire commented on Feb 28, 2024

vincentbernat commented on Feb 28, 2024

pkch commented on Oct 20, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions