Repository navigation
asyncio debug mode is very slow #76151
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2017-11-07.16:51:22.516>
created_at = <Date 2017-11-07.13:15:04.297>
labels = ['3.7', 'type-bug', 'library', 'expert-asyncio']
title = 'asyncio debug mode is very slow'
updated_at = <Date 2017-11-08.13:45:41.420>
user = 'https://github.com/pitrou'bugs.python.org fields:
activity = <Date 2017-11-08.13:45:41.420>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2017-11-07.16:51:22.516>
closer = 'pitrou'
components = ['Library (Lib)', 'asyncio']
creation = <Date 2017-11-07.13:15:04.297>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 31970
keywords = ['patch']
message_count = 13.0
messages = ['305752', '305757', '305777', '305780', '305781', '305784', '305834', '305836', '305837', '305838', '305839', '305840', '305841']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'giampaolo.rodola', 'asvetlov', 'yselivanov']
pr_nums = ['4314', '4322']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31970'
versions = ['Python 3.6', 'Python 3.7']
Activity
pitrou commentedon Nov 7, 2017
Each time a new future, handle or task is created, asyncio with debug mode enabled will parse the whole call stack and create a StackSummary object for it. Imagine a recursive coroutine: with N nested calls, you get O(N**2) performance.
Ideally debug mode wouldn't slow things too much, at least for development and testing setups.
pitrou commentedon Nov 7, 2017
If this is accepted on the principle, it would be great to also patch 3.6.
pitrou commentedon Nov 7, 2017
New changeset 921e943 by Antoine Pitrou in branch 'master':
bpo-31970: Reduce performance overhead of asyncio debug mode. (bpo-4314)
921e943
pitrou commentedon Nov 7, 2017
New changeset d8d218f by Antoine Pitrou in branch '3.6':
[3.6] bpo-31970: Reduce performance overhead of asyncio debug mode. (GH-4314) (bpo-4322)
d8d218f
pitrou commentedon Nov 7, 2017
Thank you for the quick reviews!
1st1 commentedon Nov 7, 2017
Thank you for helping with asyncio! I'll try to get to that transport performance issues you found sometime this week. I've a few ideas how to add 0-copy support to protocols.
pitrou commentedon Nov 8, 2017
I'd be interesting to hear about them. The main challenge IMHO is to find a way to allow a readinto()-like functionality.
giampaolo commentedon Nov 8, 2017
Exposing sendfile() should be straightforward. I started implementing it years ago but I gave up pretty soon because asyncio had no solid test framework for testing an actual data transfer between two sockets and that basically represented a blocker. Basically back then all recv() / send() related tests were mocks. Not sure about the current situation but if that has changed I would be happy to contribute a PR (I was the one who contributed socket.sendfile()).
As for a readinto()-like functionality: the only thing I'm aware of is splice() syscall but it's Linux only. It must be noted that also sendfile() is UNIX only. Windows achieve the same via TransmitFile.
pitrou commentedon Nov 8, 2017
Le 08/11/2017 à 14:40, Giampaolo Rodola' a écrit :
sendfile() is not useful for data that's read from (or written to) memory.
This is not the abstraction level we are talking about. The problem is
the Protocol API imposes copies by default (data_received() gives you an
arbitrarily-sized bytes object that doesn't match how your protocol
chunks data).
giampaolo commentedon Nov 8, 2017
Sorry. It looks like I completely misunderstood. =)
vstinner commentedon Nov 8, 2017
IMHO performance is a good motivation to enhance the asyncio API ;-) The tricky part is to keep the backward compatibility.
1st1 commentedon Nov 8, 2017
Let's stop the discussion in this ticket :)
pitrou commentedon Nov 8, 2017
In any case, I think this is better discussed on async-sig (where I already started a discussion thread some weeks ago -- see """APIs for high-bandwidth large I/O?""").