tarfile in stream mode always set zlib compression level to 9 #70441
Description
| BPO | 26253 |
|---|---|
| Nosy | @gustaebel, @vadmium, @wimglenn, @zhangyangyu, @jarondl |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2016-02-01.00:55:23.398>
labels = ['3.7', 'type-feature', 'library']
title = 'tarfile in stream mode always set zlib compression level to 9'
updated_at = <Date 2017-08-01.17:03:47.325>
user = 'https://bugs.python.org/PatrikDufresne'bugs.python.org fields:
activity = <Date 2017-08-01.17:03:47.325>
actor = 'jarondl'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-02-01.00:55:23.398>
creator = 'Patrik Dufresne'
dependencies = []
files = []
hgrepos = []
issue_num = 26253
keywords = []
message_count = 6.0
messages = ['259304', '259308', '259987', '292628', '297220', '299622']
nosy_count = 6.0
nosy_names = ['lars.gustaebel', 'martin.panter', 'wim.glenn', 'xiang.zhang', 'Patrik Dufresne', 'jarondl']
pr_nums = ['2962']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26253'
versions = ['Python 3.7']
Activity
PatrikDufresne commentedon Feb 1, 2016
When using tarfile.open(mode='w|gz'), the compression level is hard-coded to 9. Seed _Stream._init_write_gz():
self.zlib.compressobj(9,
In regards to zlib, I would start by replacing the value of 9 by zlib.Z_DEFAULT_COMPRESSION. This is the default value and zipfile is using it. Why using something different.
Then, I would also love to control the compression level when calling tarfile.open()
vadmium commentedon Feb 1, 2016
It looks like the default has been hard-coded to 9 ever since tarfile was added to Python. The gzip module is also hard-coded to 9 since it was added. If tarfile is changed, maybe gzip should too.
Why would you want to use zlib’s default (apparently 6)? Memory usage or speed perhaps? If we do change the default, maybe it is best to only do it in 3.6. I don’t see it as a bug fix, and there is a chance it could break someone’s code.
To be able to control the compression level, perhaps you can already do it by wrapping the tar stream with GzipFile (untested):
If the default is changed, it certainly makes sense to add an easy compression level parameter, to be able to restore the old behaviour.
vadmium commentedon Feb 10, 2016
Actually it’s not really obvious from the signatures, but in the middle of the tarfile.open() documentation it says “. . . tarfile.open() accepts the keyword argument _compresslevel_”, so it should already be possible.
zhangyangyu commentedon Apr 30, 2017
*compresslevel* takes effect for modes 'w:gz', 'r:gz', 'w:bz2', 'r:bz2', 'x:gz', 'x:bz2'. For stream modes, 'r|gz', 'w|gz', 'r|bz2', 'w|bz2', the *compresslevel* doesn't make sense. It seems not hard to make it possible but I'm not sure it's worth it or there is any reason it's hard-coded.
wimglenn commentedon Jun 28, 2017
This issue also got me. compresslevel kwarg works fine for tarfile.open(..., mode='w:gz') but raises exception for tarfile.open(..., mode='w|gz')
I want to use stream compression, and compresslevel=1 is more than enough for my use case, the default of 9 is way too slow.
jarondl commentedon Aug 1, 2017
I have submitted a PR on GitHub #2962
tiran commentedon Jun 25, 2022
The PR broke tests on systems without the optional bz2 module.
tiran commentedon Jun 25, 2022
pythongh-70441: Fix test_tarfile on systems w/o bz2 (pythongh-2962)
gh-70441: Fix test_tarfile on systems w/o bz2 (gh-2962) (#94258)
Revert 30935 defer bpo45162 to 312 (#94285)
hauntsaninja commentedon Oct 12, 2022
Thanks, looks like this has been completed and the test has been fixed