os.fwalk() unhandled exception when error occurs accessing symbolic link target

[SamsonLee]

BPO	28353
Nosy	@loewis, @hynek, @serhiy-storchaka, @zhangyangyu, @Mariatta
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	fwalk_oserror.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-10-28.06:26:12.146>
created_at = <Date 2016-10-04.07:06:55.953>
labels = ['3.7', 'type-bug', 'library']
title = 'os.fwalk() unhandled exception when error occurs accessing symbolic link target'
updated_at = <Date 2017-03-31.16:36:09.654>
user = 'https://bugs.python.org/SamsonLee'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:09.654>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-10-28.06:26:12.146>
closer = 'xiang.zhang'
components = ['Library (Lib)']
creation = <Date 2016-10-04.07:06:55.953>
creator = 'Samson Lee'
dependencies = []
files = ['44982']
hgrepos = []
issue_num = 28353
keywords = ['patch']
message_count = 16.0
messages = ['278016', '278017', '278018', '278024', '278044', '278118', '278178', '279393', '279418', '279422', '279581', '279582', '279583', '279584', '279585', '279586']
nosy_count = 8.0
nosy_names = ['loewis', 'neologix', 'python-dev', 'hynek', 'serhiy.storchaka', 'xiang.zhang', 'Samson Lee', 'Mariatta']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28353'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[SamsonLee]

The bug is os.fwalk() crashes with unhandled exception when there is an error
accessing symbolic link targets.

To reproduce the bug, create a symbolic link that targets a file that you do not
have permission to access:

    $ touch handsoff
    $ sudo chown root:root handsoff
    $ sudo chmod 700 handsoff
    $ ln -s handsoff blah

Now, os.fwalk() fails:

    >>> for root, dirs, files, fd in os.fwalk('.'):
    ...     print(root, dirs, files)
    ...
    Traceback (most recent call last):
      File "test_fwalk_permission_error.py", line 3, in <module>
        for root, dirs, files, fd in os.fwalk('.'):
      File "/usr/lib64/python3.5/os.py", line 520, in fwalk
        yield from _fwalk(topfd, top, topdown, onerror, follow_symlinks)
      File "/usr/lib64/python3.5/os.py", line 537, in _fwalk
        if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
    PermissionError: [Errno 13] Permission denied: 'blah'

The cause of the problem is in this part of os.py:

for name in names:
    try:
        # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
        # walk() which reports symlinks to directories as directories.
        # We do however check for symlinks before recursing into
        # a subdirectory.
        if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
            dirs.append(name)
        else:
            nondirs.append(name)
    except FileNotFoundError:
        try:
            # Add dangling symlinks, ignore disappeared files
            if st.S_ISLNK(stat(name, dir_fd=topfd, follow_symlinks=False)
                        .st_mode):
                nondirs.append(name)
        except FileNotFoundError:
            continue

To fix it, simply replace FileNotFoundError with more general OSError.

Cheers

[SamsonLee]

Similar to http://bugs.python.org/issue25860

[serhiy-storchaka]

I can't reproduce the issue.

$ mkdir testdir
$ touch testdir/handsoff
$ sudo chown root:root testdir/handsoff
$ sudo chmod 700 testdir/handsoff
$ ln -s handsoff testdir/blah
$ python3.5
>>> import os
>>> list(os.walk('testdir'))
[('testdir', [], ['handsoff', 'blah'])]
>>> list(os.fwalk('testdir'))
[('testdir', [], ['handsoff', 'blah'], 3)]

Ubuntu 16.04, Linux 4.4, tested on ext4 and tmpfs filesystems.

[SamsonLee]

Sorry, the target file needs to be in a directory you don't have permission to access:

    $ mkdir handsoff
    $ sudo chown root:root handsoff
    $ sudo chmod 700 handsoff
    $ ln -s handsoff/anything blah

[serhiy-storchaka]

Yes, there is a difference between os.walk() and os.fwalk() in some cases. I don't know whether it was introduced deliberately or by accident. os.fwalk() was added in bpo-13734, FileNotFoundError is used since bpo-14773.

[SamsonLee]

This is definitely a bug because there is no way an unhandled exception like this is acceptable behaviour. The behaviour is that the fwalk iteration will stop and there is no way to recover / continue. Also fwalk takes an onerror callback function that won't be called with this error.

[serhiy-storchaka]

Here is a patch with tests. Needed to test it on Windows.

[python-dev]

New changeset fe1fea4ded04 by Serhiy Storchaka in branch '3.5':
Issue bpo-28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/fe1fea4ded04

New changeset e99ec3c77a63 by Serhiy Storchaka in branch '3.6':
Issue bpo-28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/e99ec3c77a63

New changeset 33a1a3dd0051 by Serhiy Storchaka in branch 'default':
Issue bpo-28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/33a1a3dd0051

[python-dev]

New changeset ec12e16ea6a1 by Serhiy Storchaka in branch '3.6':
Issue bpo-28353: Try to fix tests.
https://hg.python.org/cpython/rev/ec12e16ea6a1

New changeset a0913dbadea6 by Serhiy Storchaka in branch 'default':
Issue bpo-28353: Try to fix tests.
https://hg.python.org/cpython/rev/a0913dbadea6

[python-dev]

New changeset 470224ec16b6 by Serhiy Storchaka in branch '3.5':
Issue bpo-28353: Fixed tests of os.fwalk() with broken links.
https://hg.python.org/cpython/rev/470224ec16b6