TarFile.getmember on directory requires trailing slash iff over 100 chars

[moloney]

BPO	21987
Nosy	@gustaebel, @vstinner, @bitdancer, @serhiy-storchaka, @miss-islington, @afonari, @akulakov
PRs	bpo-21987: fix TarFile.getmember getting a dir with a trailing slash #30283 [3.10] bpo-21987: Fix TarFile.getmember getting a dir with a trailing slash (GH-30283) #30737 [3.9] bpo-21987: Fix TarFile.getmember getting a dir with a trailing slash (GH-30283) #30738
Files	tarfile_issue.py issue21987.diff issue21987_py3.5_with_test.patch issue21987_py2.7_with_test.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-23.17:54:59.291>
created_at = <Date 2014-07-16.03:40:59.246>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'TarFile.getmember on directory requires trailing slash iff over 100 chars'
updated_at = <Date 2022-01-23.17:54:59.290>
user = 'https://bugs.python.org/moloney'

bugs.python.org fields:

activity = <Date 2022-01-23.17:54:59.290>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-23.17:54:59.291>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2014-07-16.03:40:59.246>
creator = 'moloney'
dependencies = []
files = ['35976', '36045', '36096', '36202']
hgrepos = []
issue_num = 21987
keywords = ['patch']
message_count = 15.0
messages = ['223167', '223174', '223264', '223432', '223732', '224014', '224542', '348618', '376370', '376373', '409261', '409265', '411089', '411092', '411391']
nosy_count = 10.0
nosy_names = ['lars.gustaebel', 'vstinner', 'r.david.murray', 'serhiy.storchaka', 'puppet', 'moloney', 'zigg', 'miss-islington', 'af', 'andrei.avk']
pr_nums = ['30283', '30737', '30738']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21987'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[moloney]

If a directory path is under 100 char you have to omit the trailing slash from the name passed to 'getmember'. If it is over 100 you have to include the trailing slash.

As a work around I can use the private '_getmember' with 'normalize=True'.

I tested on 2.7.2 and searched the release notes looking for a related fix since then. I couldn't find anything there, or here in the issue tracker.

[serhiy-storchaka]

Could you please provide an example?

[moloney]

Here is a script illustrating the issue.

[bitdancer]

There is indeed special logic that triggers if the name is longer than 100 characters. Presumably it has a bug. Marking this as easy since it shouldn't be too hard, given the failure example, to figure out what is wrong and fix it (and turn the example into a unit test).

It doesn't look like the relevant code has changed in python3, so the bug probably exists there as well.

[gustaebel]

Apparently, the problem is located in TarInfo._proc_gnulong(). I attached a patch.

When tarfile reads an archive, it strips trailing slashes from all filenames, except GNUTYPE_LONGNAME headers, which is a bug. tarfile creates GNU_FORMAT tar files by default, hence it uses an additional GNUTYPE_LONGNAME header for filenames >100 chars. That's why tarfile_issue.py fails if used with PAX_FORMAT, because PAX_FORMAT doesn't have this bug.

[zigg]

Here is a 3.5 fix based on Lars Gustäbel's, with test.

[puppet]

Added Matt Behrens test to Lars Gustäbel 2.7 version.

[vstinner]

This issue is 5 years old has 4 patches: it's far from being "newcomer friendly", I remove the "Easy" label.

[afonari]

Any updates on this?

[vstinner]

Any updates on this?

So far, nobody proposed a pull request. So no, there is no update.

Someone has to step in, dig into the issue, propose a fix, then someone else has to review the PR, and finally the PR should be merged.

[akulakov]

The original issue was twofold:

1. below 100 char not working with trailing slash
2. over 100 char not working WITHOUT trailing slash

The second part is no longer an issue -- tested in 3.9 and 3.11 on MacOS.

Currently the issue is that a trailing slash now doesn't work for lookup of dirs, no matter the size of name.

This is inconsistent with the way shell commands work as well as various Python path related modules that tolerate trailing slash for dirs.

This can cause users to wrongly assume a dir is absent in a tarfile, so I think it's worth fixing and I've added a PR with a test for both old and new issue.