distutils.filelist.findall() fails on broken symlink in Py2.x

[AlexanderDutton]

BPO	12885
Nosy	@jaraco, @tarekziade, @merwok, @vadmium, @dstufft
Files	faf37fc3b097.diff 3da42b593e1f.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = <Date 2016-09-02.01:27:07.588>
created_at = <Date 2011-09-02.13:57:22.428>
labels = ['type-bug', 'library']
title = 'distutils.filelist.findall() fails on broken symlink in Py2.x'
updated_at = <Date 2016-09-06.02:29:57.112>
user = 'https://bugs.python.org/AlexanderDutton'

bugs.python.org fields:

activity = <Date 2016-09-06.02:29:57.112>
actor = 'python-dev'
assignee = 'jaraco'
closed = True
closed_date = <Date 2016-09-02.01:27:07.588>
closer = 'jaraco'
components = ['Distutils']
creation = <Date 2011-09-02.13:57:22.428>
creator = 'Alexander.Dutton'
dependencies = []
files = ['40303', '40516']
hgrepos = ['315']
issue_num = 12885
keywords = ['patch']
message_count = 16.0
messages = ['143399', '143413', '143496', '143534', '230021', '249366', '249384', '251083', '251089', '251170', '274191', '274193', '274195', '274197', '274199', '274501']
nosy_count = 8.0
nosy_names = ['jaraco', 'tarek', 'eric.araujo', 'alexis', 'python-dev', 'martin.panter', 'Alexander.Dutton', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12885'
versions = ['Python 3.5', 'Python 3.6']

[AlexanderDutton]

If there are any broken symlinks in the same directory as a setup.py when e.g. sdist is run, findall() will fall over when attempting to os.stat() the symlink:

Traceback (most recent call last):
  File "setup.py", line 81, in run
    _sdist.run(self)
  File "/usr/lib/python2.6/distutils/command/sdist.py", line 144, in run
    self.get_file_list()
  File "/usr/lib/python2.6/distutils/command/sdist.py", line 238, in get_file_list
    self.filelist.findall()
  File "/usr/lib/python2.6/distutils/filelist.py", line 47, in findall
    self.allfiles = findall(dir)
  File "/usr/lib/python2.6/distutils/filelist.py", line 297, in findall
    stat = os.stat(fullname)
OSError: [Errno 2] No such file or directory: 'debian/tmp/usr/share/somepath/somesymlink'

Solutions would include replacing the call to os.stat() with one to os.lstat() (probably backwards-incompatible), or trying one and then the other.

This bug is present in Pythons 2.6.6 (Debian 6.0.2) and 2.7 (Fedora 14).

When attempting to reproduce in Python 3.1.2 (on Fedora) no error was encountered. However, looking at distutils/filelist.py, the same unadulterated call to os.stat() is present. I'll presume that for whatever reason distutils in Py3.1.2 never has cause to stat my broken symlink.

[merwok]

Symlinks are barely supported by distutils; the only mention you can find in the doc is that the MANIFEST file can contain symlinks. In the light of that, one could argue that broken symlinks are not supported nor ignored, and that you just should not have them. Is there a valid use case for having broken symlinks in a Python project?

(About versions: I’m removing 2.6 which only gets security fixes, and adding 3.x to add regression tests to them, even if the bug isn’t there now.)

[AlexanderDutton]

I've come across it as I'm creating a Debian package of the Python package in the same tree — I'm happy to be told this is a Bad Idea and that they should be in different places.

The broken symlinks are relative and in debian/tmp, and will point to locations provided by other Debian packages once my package is installed in the right location.

FWIW, I'm getting round it at the moment by walking the directory tree and removing the files is os.path.islink(filename) and not os.path.exists(os.path.join(filename, os.readlink(filename))).

I'm happy to provide tests and a patch if necessary.

[merwok]

I've come across it as I'm creating a Debian package of the Python
package in the same tree

I think a lot of people are doing this.

The broken symlinks are relative and in debian/tmp, and will point to
locations provided by other Debian packages once my package is
installed in the right location.

It’s too bad that filelist goes into the debian subdirectory :( The Ubuntu-originated python-distutils-extra project had a similar problem and they switched from FileList.findall to os.walk to avoid it.

What happens if you ignore the debian dir in MANIFEST.in?

[jaraco]

This issue has been monkeypatched by setuptools for 7 years: https://bitbucket.org/pypa/setuptools/commits/4556f9d08ef7

[jaraco]

Eric, would you be willing to review the commits in the referenced patch repo (collapsed in the attached diff)? If you're happy with the implementation, I'll backport it to Python 2.7 and apply it to 3.4-3.6 and then update setuptools to only monkeypatch older distutils.

[merwok]

Patch looks good and would fix the reported problem. Being a backport from setuptools also gives confidence.

Does the patch change the behaviour for the handling of the MANIFEST file (not MANIFEST.in)? My previous message mentions that the docs say that one can include symlinks in MANIFEST.