Reference cycle in email.feedparser

[Costas]

BPO	27522
Nosy	@warsaw, @bitdancer
Files	emailfeedparsergc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-07-16.01:31:42.564>
created_at = <Date 2016-07-15.19:17:54.962>
labels = ['expert-email', 'performance']
title = 'Reference cycle in email.feedparser'
updated_at = <Date 2016-07-16.01:31:58.416>
user = 'https://bugs.python.org/Costas'

bugs.python.org fields:

activity = <Date 2016-07-16.01:31:58.416>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2016-07-16.01:31:42.564>
closer = 'r.david.murray'
components = ['email']
creation = <Date 2016-07-15.19:17:54.962>
creator = 'Costas'
dependencies = []
files = ['43733']
hgrepos = []
issue_num = 27522
keywords = ['patch']
message_count = 3.0
messages = ['270503', '270527', '270528']
nosy_count = 4.0
nosy_names = ['barry', 'r.david.murray', 'python-dev', 'Costas']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue27522'
versions = ['Python 3.5', 'Python 3.6']

[Costas]

Background: We have two (clusters of) servers running a high-throughput websocket application. One runs under Python 2.7, the other has been migrated to 3.4. Very similar code runs on both servers, and both manually call the gc periodically to reduce latency.

Problem: We noticed that under load the 3.4 server seems to slowly "leak" memory between gc collections, while the 2.7 server doesn't have this issue.

Investigation: I tracked down the source of all the garbage to email.feedparser, which is used by http.server to parse the headers of incoming requests. The initialiser of the FeedParser class creates a reference cycle by putting a lambda function into a member variable. The lambda contains a closure with its surrounding scope that points to the object, and the object points back to the lambda.

Proposed fix: The lambda can be harmlessly replaced by a boolean value, which doesn't cause the problem, and it's much more readable too. I'm attaching a little patch.

[python-dev]

New changeset 6c7fd035bce3 by R David Murray in branch '3.5':
bpo-27522: break unintended cycle in feedparser.
https://hg.python.org/cpython/rev/6c7fd035bce3

New changeset e1278508f3cb by R David Murray in branch 'default':
Merge: bpo-27522: break unintended cycle in feedparser.
https://hg.python.org/cpython/rev/e1278508f3cb

[bitdancer]

Thanks, Costas.