Repository navigation
ConfigParser 50x slower in 2.7 #55298
Description
| BPO | 11089 |
|---|---|
| Nosy | @smontanaro, @birkenfeld, @rhettinger, @skrah, @ambv |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/rhettinger'
closed_at = <Date 2011-02-21.19:43:02.665>
created_at = <Date 2011-02-01.14:04:42.356>
labels = ['extension-modules', 'performance']
title = 'ConfigParser 50x slower in 2.7'
updated_at = <Date 2011-02-21.19:43:02.664>
user = 'https://bugs.python.org/vlachoudis'bugs.python.org fields:
activity = <Date 2011-02-21.19:43:02.664>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2011-02-21.19:43:02.665>
closer = 'rhettinger'
components = ['Extension Modules']
creation = <Date 2011-02-01.14:04:42.356>
creator = 'vlachoudis'
dependencies = []
files = ['20640', '20643', '20654']
hgrepos = []
issue_num = 11089
keywords = ['patch']
message_count = 9.0
messages = ['127699', '127710', '127713', '127715', '127726', '127761', '127764', '127766', '128986']
nosy_count = 6.0
nosy_names = ['skip.montanaro', 'georg.brandl', 'rhettinger', 'skrah', 'lukasz.langa', 'vlachoudis']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue11089'
versions = ['Python 3.2']
Activity
vlachoudis commentedon Feb 1, 2011
The ConfigParser class in 2.7 is almost >50 times slower than in the 2.6 which for large files it renders it almost unusable. Actually the speed decrease depends on the amount of the stored data
Results from test program:
Python 2.7 (r27:82500, Sep 16 2010, 18:02:00)
on 3.5GHz Fedora14 64bit machine
ConfigParser 166.307140827
RawConfigParser 0.1887819767
Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:31)
on 3.0GHz Fedora13 64bit machine
ConfigParser 4.24494099617
RawConfigParser 0.172905921936
skrah commentedon Feb 1, 2011
If OrderedDict is used, the test case quickly uses 8GB of memory. With
this change (I'm not suggesting this as a fix!), the timings are normal:
Index: Lib/ConfigParser.py
===================================================================
rhettinger commentedon Feb 2, 2011
Commenting-out the
c.set(section,"item#%d"%(i),str(i))calls shows that that is not where the problem lies for the ConfigParser() class.The issue seems confined to ConfigParser.get().
The RawConfigParser() class doesn't seem to have the same issue.
Looking at the 2.7 code for ConfigParser.get() shows that it is doing a lot more than just getting. For example, it does a full copy of defaults dictionary on every call !?
I'll look at it more shortly.
rhettinger commentedon Feb 2, 2011
Attaching a patch that fixes the algorithmic atrocities by using the Chainmap recipe:
http://code.activestate.com/recipes/305268-chained-map-lookups
rhettinger commentedon Feb 2, 2011
Fixed for 2.7 in r88318. Will make a similar fix for 3.1.4 and for 3.2.1.
rhettinger commentedon Feb 2, 2011
Attaching patch for Python 3.2.
Georg, I was think of waiting for 3.2.1 for this one, but it can go into 3.2.0 RC2 if you prefer.
birkenfeld commentedon Feb 2, 2011
3.2.1 should be fine.
rhettinger commentedon Feb 2, 2011
Fixed 3.1 in r88323.
rhettinger commentedon Feb 21, 2011
See r88469 and r88470.