LZMACompressor and LZMADecompressor raise exceptions if given empty strings twice

[benfogle]

BPO	27517
Nosy	@serhiy-storchaka, @AraHaan, @benfogle
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	lzma.patch lzma_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-10-31.06:36:55.434>
created_at = <Date 2016-07-15.01:03:30.680>
labels = ['extension-modules', 'type-bug', '3.7']
title = 'LZMACompressor and LZMADecompressor raise exceptions if given empty strings twice'
updated_at = <Date 2017-03-31.16:36:25.500>
user = 'https://github.com/benfogle'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:25.500>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-10-31.06:36:55.434>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2016-07-15.01:03:30.680>
creator = 'benfogle'
dependencies = []
files = ['43727', '45286']
hgrepos = []
issue_num = 27517
keywords = ['patch']
message_count = 7.0
messages = ['270451', '270452', '270455', '279737', '279758', '279764', '279765']
nosy_count = 5.0
nosy_names = ['nadeem.vawda', 'python-dev', 'serhiy.storchaka', 'Decorater', 'benfogle']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27517'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[benfogle]

To reproduce:

>>> import lzma
>>> c = lzma.LZMACompressor()
>>> c.compress(b'')
b'\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F'
>>> c.compress(b'')
b''
>>> c.compress(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_lzma.LZMAError: Insufficient buffer space
>>> d = lzma.LZMADecompressor()
>>> d.decompress(b'')
b''
>>> d.decompress(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_lzma.LZMAError: Insufficient buffer space

This can occur anytime during compression/decompression when an empty string is passed twice in a row.

The problem is that that liblzma interprets a call to lzma_code() with avail_in == 0 as a buffer full condition. The second time in a row it encounters this, it returns LZMA_BUF_ERROR as per documentation. The attached patch prevents this condition from occurring.

[AraHaan]

Why you passing decompress without passing compress before decompressing it again?

Also I would expect that it would show the same compress result trice in a row on the 1st test. This must definately be a issue.

[benfogle]

The above code demonstrates two separate issues. One with the decompressor, and one with the compressor.

In the compressor example, the first output differs from the rest because it is a file header which is always emitted. That behavior is correct.

[serhiy-storchaka]

Thank you for your report and patch Benjamin. Seems your patch fixes the problem in default case. But the compressor still fails with the raw format.

>>> import lzma
>>> FILTERS_RAW_4 = [{"id": lzma.FILTER_DELTA, "dist": 4},
...                  {"id": lzma.FILTER_X86, "start_offset": 0x40},
...                  {"id": lzma.FILTER_LZMA2, "preset": 4, "lc": 2}]
>>> c = lzma.LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
>>> c.compress(b'')
b''
>>> c.compress(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_lzma.LZMAError: Insufficient buffer space

[benfogle]

Ah, thank you. Good catch. I have reworked the patch to handle both cases.

[python-dev]

New changeset b06f15507978 by Serhiy Storchaka in branch '3.5':
Issue bpo-27517: LZMA compressor and decompressor no longer raise exceptions if
https://hg.python.org/cpython/rev/b06f15507978

New changeset fb64c7a81010 by Serhiy Storchaka in branch '3.6':
Issue bpo-27517: LZMA compressor and decompressor no longer raise exceptions if
https://hg.python.org/cpython/rev/fb64c7a81010

New changeset 98c078fca8e0 by Serhiy Storchaka in branch 'default':
Issue bpo-27517: LZMA compressor and decompressor no longer raise exceptions if
https://hg.python.org/cpython/rev/98c078fca8e0

[serhiy-storchaka]

Thank you for you contribution Benjamin.