tarfile add uses random order

[bmwiedemann]

BPO	30693
Nosy	@rhettinger, @gustaebel, @vstinner, @tiran, @ned-deily, @bitdancer, @serhiy-storchaka, @bmwiedemann
PRs	bpo-30693: zip+tarfile: sort directory listing #2263 bpo-30693: Fix tarfile test cleanup on MSWindows #5557 [3.7] bpo-30693: Fix tarfile test cleanup on MSWindows (GH-5557) #5567 bpo-46951: Order contents of zipapps #31713

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-02-06.18:38:13.647>
created_at = <Date 2017-06-18.02:03:30.834>
labels = ['3.7', 'type-bug', 'library']
title = 'tarfile add uses random order'
updated_at = <Date 2022-03-06.22:48:55.393>
user = 'https://github.com/bmwiedemann'

bugs.python.org fields:

activity = <Date 2022-03-06.22:48:55.393>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2018-02-06.18:38:13.647>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2017-06-18.02:03:30.834>
creator = 'bmwiedemann'
dependencies = []
files = []
hgrepos = []
issue_num = 30693
keywords = ['patch']
message_count = 23.0
messages = ['296251', '296280', '296308', '310164', '310166', '310167', '310169', '310170', '310171', '310173', '310205', '310337', '310342', '311318', '311320', '311321', '311324', '311325', '311387', '311605', '311685', '311736', '311738']
nosy_count = 9.0
nosy_names = ['rhettinger', 'lars.gustaebel', 'vstinner', 'christian.heimes', 'ned.deily', 'r.david.murray', 'python-dev', 'serhiy.storchaka', 'bmwiedemann']
pr_nums = ['2263', '5557', '5567', '31713']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30693'
versions = ['Python 3.7']

[bmwiedemann]

Filesystems do not give any guarantees about ordering of files returned in directory listings, thus tarfile.add adds files in random order, when using os.listdir in recursion.

See also https://reproducible-builds.org/docs/stable-inputs/ on that topic.

[serhiy-storchaka]

The patch for similar issue with the glob module was rejected recently since it is easy to sort the result of glob.glob() (see bpo-30461). This issue looks similar, but there are differences. On one side, the command line tar utility doesn't have the option for sorting file names and seems don't sort them by default (I didn't checked). It is possible to use external sorting with the tarfile module as with the tar utility (generate the list of all files and directories, sort it, and pass every item to TarFile.add with the option recursive=False). But on other side, this is not so easy as for glob.glob(). And the overhead of the sorting is expected to be smaller than for glob.glob(). This may be considered as additional arguments for approving the patch.

If this approach will be approved, it should be applied also to the ZIP archives.

FYI the order of archived files can affect the compression ratio of the compressed tar archive. For example the 7-Zip archiver sorts files by extensions, this increases the chance that files of the same type (text, multimedia, spreadsheet, executables, etc) are grouped together and use the common dictionary for global compression. This isn't directly related to this issue, just a material for possible future enhancement.

[bmwiedemann]

note: recent GNU tar versions (1.28?) added an option --sort=name

and the overhead of sorting (e.g. I measured 4ms for 10000 files) is negligible compared to the other processing done on the files here.

[bitdancer]

Given the reproducible builds angle, I'd say this was worth doing.

[tiran]

+1 from me

In my opinion it's both a good idea to not sort the result of glob.glob() and make the order of tar and zip module content ordered. The glob module is low level and it makes sense to expose the file system sort order.

On the other hand tar and zip modules are on a higher level. Without sorting it's impossible to create reproducible archives. The performance impact is irrelevant. I/O and compression dominant performance.

[tiran]

PS: I'm -0 to backport the change to 3.6 and 2.7. 3.5 is in security fix mode and therefore completely out of scope.

[vstinner]

Since we currently don't warranty *anything* about ordering, I like the idea of *fixing* Python 2.7 and 3.6 as well. +1 for fix it in 2.7, 3.6 and master.

[bitdancer]

Ah, I was just going to ask about that. I guess I'm -0 on the backport as well. The other reproducible build stuff is only going to land in 3.7. However, this is in a more general category than the pyc stuff, so I can see the argument for backporting it.

[vstinner]

The only warranty in that TarFile.getmembers(), TarFile.getnames() and ZipFile.infolist() returns members/names "in the same order as the members in the archive".

Currently, there is no warranty when packing, only on unpack.

[tiran]

The patch changes behavior. It's fine for 3.7 but not for 3.6/2.7. Somebody may depend on filesystem order.

[ned-deily]

This doesn't seem appropriate to me for backporting to existing releases (3.6. and 2.7). AFAIK, the current file-system-order behavior has never been identified as a bug. Unless there is a stronger case for changing the existing 3.6.x behavior, I am -1 on backporting.

[serhiy-storchaka]

If make this change you need to make similar changes in other places that recursively add files to archives: shutil, zipapp, distutils, and maybe more.