os.fwalk() silently skips remaining directories when error occurs

[SamsonLee]

BPO	25860
Nosy	@vstinner, @benhoyt, @hynek, @serhiy-storchaka
Files	fwalk-silently-skips-dirs-when-error-occurs.patch fwalk-silently-skips-dirs-when-error-occurs-with-tests.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-12-23.09:31:10.785>
created_at = <Date 2015-12-14.10:30:55.596>
labels = ['type-bug', 'library']
title = 'os.fwalk() silently skips remaining directories when error occurs'
updated_at = <Date 2015-12-23.09:31:10.785>
user = 'https://bugs.python.org/SamsonLee'

bugs.python.org fields:

activity = <Date 2015-12-23.09:31:10.785>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-12-23.09:31:10.785>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-12-14.10:30:55.596>
creator = 'Samson Lee'
dependencies = []
files = ['41304', '41357']
hgrepos = []
issue_num = 25860
keywords = ['patch']
message_count = 4.0
messages = ['256377', '256699', '256871', '256876']
nosy_count = 7.0
nosy_names = ['vstinner', 'benhoyt', 'neologix', 'python-dev', 'hynek', 'serhiy.storchaka', 'Samson Lee']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25860'
versions = ['Python 3.5', 'Python 3.6']

[SamsonLee]

I noticed os.fwalk() produced different results as os.walk(). It turned out
that os.fwalk() skips all remaining directories when OSError occurs (e.g. due
to PermissionError).

To reproduce the bug, first create a test directory structure:

    $ mkdir 1; touch 1/a
    $ mkdir 2; touch 2/b
    $ mkdir 3; touch 3/c
    $ mkdir 4; touch 4/c

At this stage, everything is okay, both os.fwalk() os.walk() produce the same
results:

    >>> import os
    >>> for root, dirs, files in os.walk('.'):
    ...     dirs.sort()
    ...     print(root, dirs, files)
    . ['1', '2', '3', '4'] []
    ./1 [] ['a']
    ./2 [] ['b']
    ./3 [] ['c']
    ./4 [] ['d']
    >>> for root, dirs, files, fd in os.fwalk('.'):
    ...     dirs.sort()
    ...     print(root, dirs, files)
    . ['1', '2', '3', '4'] []
    ./1 [] ['a']
    ./2 [] ['b']
    ./3 [] ['c']
    ./4 [] ['d']

To introduce an error, force a PermissionError on one of the directories:

    $ sudo chown root:root 2
    $ sudo chmod 700 2

Now, the os.fwalk() results are different (trust me, os.walk() is still okay):

    >>> for root, dirs, files, fd in os.fwalk('.'):
    ...     dirs.sort()
    ...     print(root, dirs, files)
    . ['1', '2', '3', '4'] []
    ./1 [] ['a']

So it seems that os.fwalk skips remaining directories after an error occurs.

The cause of the problem is in this part of os.py:

    for name in dirs:
        try:
            orig_st = stat(name, dir_fd=topfd, follow_symlinks=follow_symlinks)
            dirfd = open(name, O_RDONLY, dir_fd=topfd)
        except OSError as err:
            if onerror is not None:
                onerror(err)
            break

To fix it, simply replace break with continue. Patch attached.

Cheers

[serhiy-storchaka]

LGTM. Here is a patch with tests.

[python-dev]

New changeset 767262c149ca by Serhiy Storchaka in branch '3.5':
Issue bpo-25860: os.fwalk() no longer skips remaining directories when error occurs.
https://hg.python.org/cpython/rev/767262c149ca

New changeset a85675dabb8f by Serhiy Storchaka in branch 'default':
Issue bpo-25860: os.fwalk() no longer skips remaining directories when error occurs.
https://hg.python.org/cpython/rev/a85675dabb8f

[python-dev]

New changeset 7995a81236b6 by Serhiy Storchaka in branch '3.5':
Issue bpo-25860: Fixed test failure caused by inconsistency of os.walk() and
https://hg.python.org/cpython/rev/7995a81236b6

New changeset dcf9e9ae5393 by Serhiy Storchaka in branch 'default':
Issue bpo-25860: Fixed test failure caused by inconsistency of os.walk() and
https://hg.python.org/cpython/rev/dcf9e9ae5393