zlib.error: Error -2 while flushing: inconsistent stream state

[JeremyHeiner]

BPO	30017
Nosy	@takluyver, @vadmium, @serhiy-storchaka, @zhangyangyu, @gotyaoi
PRs	bpo-30017: Allowed calling the close() method of the zip entry writer object #1041 [3.6] bpo-30017: Allowed calling the close() method of the zip entry writer object (GH-1041) #1092

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2017-04-16.09:05:54.988>
created_at = <Date 2017-04-07.14:21:06.591>
labels = ['3.7', 'type-bug', 'library']
title = 'zlib.error: Error -2 while flushing: inconsistent stream state'
updated_at = <Date 2017-04-16.09:05:54.987>
user = 'https://bugs.python.org/JeremyHeiner'

bugs.python.org fields:

activity = <Date 2017-04-16.09:05:54.987>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2017-04-16.09:05:54.988>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2017-04-07.14:21:06.591>
creator = 'Jeremy Heiner'
dependencies = []
files = []
hgrepos = []
issue_num = 30017
keywords = []
message_count = 6.0
messages = ['291275', '291309', '291310', '291319', '291550', '291747']
nosy_count = 6.0
nosy_names = ['takluyver', 'martin.panter', 'serhiy.storchaka', 'xiang.zhang', 'Ellison Marks', 'Jeremy Heiner']
pr_nums = ['1041', '1092']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30017'
versions = ['Python 3.6', 'Python 3.7']

[JeremyHeiner]

I had some statements inside a with statement to write data to an entry in a
ZipFile. It worked great. I added a second with statement containing almost
exactly the same statements. That still worked great.

I refactored those common statements into a function and called that function
from the two with statements... and got an exception:

zlib.error: Error -2 while flushing: inconsistent stream state

I can't figure out why it matters whether the writing happens in the with or
in the function called by the with, but here's a trimmed-down version of the
code that demonstrates the problem:

--------------------------------------------------------------------------------
#!/usr/bin/env python

import io, pprint, zipfile
from zipfile import ZIP_DEFLATED

def printLiteral( data, out ) :
        encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
        pprint.pprint( data, stream=encoder )

data = { 'not' : 'much', 'just' : 'some K \N{RIGHTWARDS WHITE ARROW} V pairs' }

with zipfile.ZipFile( 'zzz.zip', mode='w', compression=ZIP_DEFLATED ) as myzip :

    with myzip.open( 'this one works', 'w' ) as out :
        encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
        pprint.pprint( data, stream=encoder )

    with myzip.open( 'this one fails', 'w' ) as out :
        printLiteral( data, out )

        print( 'printed but entry still open' )
    print( 'entry has been closed but not file' )
print( 'zip file has been closed' )

---

And here's the output on my Arch Linux 64bit with package python 3.6.0-2...
A co-worker sees the same behavior on MacOS 10.11.6 Python 3.6.1 :

--------------------------------------------------------------------------------

printed but entry still open
Traceback (most recent call last):
  File "zzz.py", line 21, in <module>
    print( 'printed but entry still open' )
  File "/usr/lib/python3.6/zipfile.py", line 995, in close
    buf = self._compressor.flush()
zlib.error: Error -2 while flushing: inconsistent stream state

---

I tried debugging this in PyDev but got lost. Turning off the compression makes
the exception go away.

[gotyaoi]

At a guess, when encoder goes out of scope, it closes the underlying file object. Then, on exiting the with block, the zipfile tries to take some action with the closed file and errors out?

[vadmium]

It looks like the zip entry writer object may not expect its “close” method to be called multiple times. Other file objects tend to allow this with no ill effect. Adding Serhiy and Thomas who implemented the writer object in bpo-26039.

The first time it is called will be when the context manager (“with” statement) exits.

The second time will be when the TextIOWrapper is garbage-collected. This is documented behaviour <https://docs.python.org/3.6/library/io.html#io.IOBase.\_\_del__\>, but I think it is a design mistake of the IOBase hierarchy; see bpo-19829.

A reasonable workaround would be to call encoder.detach() after you have finished writing the zip entry.

[serhiy-storchaka]

I agree, that close() should be an idempotent operation. Proposed patch fixes this.

[serhiy-storchaka]

New changeset 4c0d9ea by Serhiy Storchaka in branch 'master':
bpo-30017: Allowed calling the close() method of the zip entry writer object (bpo-1041)
4c0d9ea

[serhiy-storchaka]

New changeset 8e5b52a by Serhiy Storchaka in branch '3.6':
bpo-30017: Allowed calling the close() method of the zip entry writer object (bpo-1041) (bpo-1092)
8e5b52a