reduce tarfile memory footprint

[gustaebel]

BPO	2058
Nosy	@gustaebel
Files	tarfile-memory.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gustaebel'
closed_at = <Date 2008-04-14.10:08:23.289>
created_at = <Date 2008-02-10.11:44:02.057>
labels = ['library', 'performance']
title = 'reduce tarfile memory footprint'
updated_at = <Date 2008-04-14.10:08:23.287>
user = 'https://github.com/gustaebel'

bugs.python.org fields:

activity = <Date 2008-04-14.10:08:23.287>
actor = 'lars.gustaebel'
assignee = 'lars.gustaebel'
closed = True
closed_date = <Date 2008-04-14.10:08:23.289>
closer = 'lars.gustaebel'
components = ['Library (Lib)']
creation = <Date 2008-02-10.11:44:02.057>
creator = 'lars.gustaebel'
dependencies = []
files = ['9399']
hgrepos = []
issue_num = 2058
keywords = ['patch']
message_count = 2.0
messages = ['62248', '65462']
nosy_count = 1.0
nosy_names = ['lars.gustaebel']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue2058'
versions = ['Python 3.0']

[gustaebel]

tarfile.py wastes lots of memory resources. The memory consumption does
not depend on the size of an archive but on the numbers of members in it.
The attached patch reduces memory usage by about 60% and consists of two
independent strategies (each with about 30% reduction):

1. Add __slots__ to the TarInfo class. This was proposed in bpo-1540385
   a while ago but rejected due to backward-compatibility issues.

2. Remove the undocumented buf attribute of the TarInfo class. buf
   stores the original 512-byte header block read from the archive. This
   was introduced in r45954 and is rather useless except for GNUTYPE_SPARSE
   processing. This might as well be a candidate for backporting to 2.6.

[gustaebel]

Checked into the py3k branch as r62337.