tar.extractall() does not recognize unexpected EOF

[ThomasGttler]

BPO	24259
Nosy	@gustaebel, @ethanfurman, @vadmium
Files	01-issue24259-test.diff tar_which_is_cut.tar issue24259-3.x.diff issue24259-2.x.diff issue24259-3.x-2.diff issue24259-2.x-2.diff issue24259-3.x-3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gustaebel'
closed_at = <Date 2015-07-06.07:34:50.217>
created_at = <Date 2015-05-21.15:18:16.111>
labels = ['type-bug', 'library']
title = 'tar.extractall() does not recognize unexpected EOF'
updated_at = <Date 2015-07-06.07:34:50.217>
user = 'https://bugs.python.org/ThomasGttler'

bugs.python.org fields:

activity = <Date 2015-07-06.07:34:50.217>
actor = 'lars.gustaebel'
assignee = 'lars.gustaebel'
closed = True
closed_date = <Date 2015-07-06.07:34:50.217>
closer = 'lars.gustaebel'
components = ['Library (Lib)']
creation = <Date 2015-05-21.15:18:16.111>
creator = 'Thomas G\xc3\xbcttler'
dependencies = []
files = ['39528', '39531', '39543', '39544', '39579', '39580', '39837']
hgrepos = []
issue_num = 24259
keywords = ['patch']
message_count = 21.0
messages = ['243755', '244152', '244233', '244235', '244236', '244237', '244239', '244248', '244254', '244282', '244287', '244289', '244290', '244360', '244361', '244433', '244455', '244572', '245584', '245979', '246350']
nosy_count = 6.0
nosy_names = ['guettli', 'lars.gustaebel', 'ethan.furman', 'python-dev', 'martin.panter', 'Thomas G\xc3\xbcttler']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24259'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[ThomasGttler]

The Python tarfile library does not detect a broken tar.

user@host$ wc -c good.tar
143360 good.tar

user@host$ head -c 130000 good.tar > cut.tar

user@host$ tar -tf cut.tar
...
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Very nice, the command line tool recognizes an unexpected EOF.

user@host$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
>>> import tarfile
>>> tar=tarfile.open('cut.tar')
>>> tar.extractall()

Not nice. The Python library decodes the file, but raises no exception.

Is this a bug or feature?

Source: http://stackoverflow.com/questions/30302204/tar-extractall-does-not-recognize-unexpected-eof

[guettli]

Who has enough knowledge of the tarfile module to create a good patch?

[vadmium]

I might be able to make a patch, but what should the patch do exactly?

• Raise an exception as soon as something wrong is found
• Defer exceptions until close() is called, to allow partial data recovery
• Add some sort of defects API that you can optionally inspect, like the email message library has
• A new tarfile.open(strict=True) mode

[vadmium]

Actually, looking closer at the module, perhaps you just need to set the errorlevel=1 option:

>>> with tarfile.open("truncated.tar", errorlevel=1) as tar:...     tar.extractall("test-dir")
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/proj/python/cpython/Lib/tarfile.py", line 1996, in extractall
    numeric_owner=numeric_owner)
  File "/home/proj/python/cpython/Lib/tarfile.py", line 2038, in extract
    numeric_owner=numeric_owner)
  File "/home/proj/python/cpython/Lib/tarfile.py", line 2108, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/home/proj/python/cpython/Lib/tarfile.py", line 2154, in makefile
    copyfileobj(source, target, tarinfo.size)
  File "/home/proj/python/cpython/Lib/tarfile.py", line 242, in copyfileobj
    raise OSError("end of file reached")
OSError: end of file reached

[ethanfurman]

On the SO question [1] the OP stated that he tried errorlevel of both 1 and 2 with no effect... however, he was using Python2.6.

Martin, can you run that same test with 2.6 to verify that errorlevel did not work there, but does now?

[1] http://stackoverflow.com/q/30302204/208880

[ethanfurman]

Actually, the OP was using 2.7.6.

[ethanfurman]

I ran the OP's code in 2.7, 3.3, 3.4, and 3.5, as well as using ubuntu's gnu tar 1.27.1:

gnu tar did not report any errors.

Python (all tested versions) did not report any errors (with the errorlevel parameter missing, set to 1, and set to 2).

[vadmium]

I guess it depends on the particular tar file and where it gets truncated. Just now I tested with a tar file created from Python’s Tools directory:

$ tar c Tools/ > good.tar
$ ls -gG good.tar 
-rw-r--r-- 1 17397760 May 28 02:43 good.tar
$ head -c 130000 good.tar > cut.tar
$ tar -tf cut.tar
Tools/
[. . .]
Tools/hg/hgtouch.py
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
[Exit 2]

Using Python 2.7.9, I can reproduce Thomas’s behaviour. It extracts the files with no error indication, and actually truncated the last extracted file to 2512 bytes. Adding errorlevel=1 or errorlevel=2 did not make a difference.

Then I tried Python 3.3, 3.4 and 3.6 in various forms. They all seem to raise OSError, even with no errorlevel specified, and create the last file with zero bytes. So in this case I think it is a Python 2 bug (which I’m personally not so motivated to fix).

Ethan: how did you create and truncate your tar file? It almost sounds like it is intact, or maybe didn’t get truncated enough to completely wipe out the EOF block.

[ethanfurman]

I took an existing tar file and chopped it in half with head -c. I was able to extract half the files, but I didn't check the viability of the last file as I was looking for tar or python error feedback.

[gustaebel]

I have written a test for the issue, so that we have a basis for discussion.

There are four different scenarios where an unexpected eof can occur: inside a metadata block, directly after a metadata block, inside a data segment or directly after a data segment (i.e. missing end of archive marker).

Case #1 is taken care of (TruncatedHeaderError).

Case #4 is merely a violation of standard, which is neglectable.

Case #2 and #3 are essentially the same. If a data segment is empty or incomplete this means data was lost when the archive was created which should not go unnoticed when reading it. (see _FileInFile.read() for the code in question)

The problem is that, even after we have fixed case #2 and #4, we have no reliable way to detect an incomplete data segment unless we read it and count the bytes. If we simply iterate over the TarFile (e.g. do a TarFile.list()) the archive will appear intact. That is because in the TarFile.next() method we seek from one metadata block to the next, but we cannot simply detect if we seek beyond the end of the archive - except if we insist on the premise that each tar that we read is standards-compliant and comes with an end of archive marker (see case #4), which we probably should not.

Three possible options come to my mind:

1. Add a warning to the documentation that in order to test the integrity of an archive the user has to read through all the data segments.
2. Instead of using seek() in TarFile.next() use read() to advance the file pointer. This is a negative impact on the performance in most cases.
3. Insist on an end of archive marker. This has the disadvantage that users may get an exception although everything is fine.