Close directory descriptor in scandir iterator on error

[serhiy-storchaka]

BPO	26117
Nosy	@gvanrossum, @vstinner, @benhoyt, @serhiy-storchaka
Files	scandir_closedir_on_error.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-02-08.15:59:36.282>
created_at = <Date 2016-01-14.22:10:22.722>
labels = ['extension-modules', 'performance']
title = 'Close directory descriptor in scandir iterator on error'
updated_at = <Date 2016-02-08.17:35:41.304>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2016-02-08.17:35:41.304>
actor = 'vstinner'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-02-08.15:59:36.282>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2016-01-14.22:10:22.722>
creator = 'serhiy.storchaka'
dependencies = []
files = ['41616']
hgrepos = []
issue_num = 26117
keywords = ['patch']
message_count = 5.0
messages = ['258232', '259850', '259851', '259856', '259866']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'vstinner', 'benhoyt', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue26117'
versions = ['Python 3.5', 'Python 3.6']

[serhiy-storchaka]

Since scandir iterator has no close method, the most safe way to avoid file descriptors leaks when use os.scandir() is accumulating yielded results into a list before processing them:

    entries = list(os.scandir(path))

But this doesn't save us from all leaks. If an error happened during yielding next entry, opened file descriptor left open.

Proposed patch makes scandir to close the file descriptor not only when an iteration is exhausted, but when any error (expected OSError or MemoryError) is raised.

This is needed to fix possible leaks in os.walk() in 3.5 (see also bpo-25995).

[serhiy-storchaka]

Could anyone please make a review?

[vstinner]

Could anyone please make a review?

Done. Enjoy :-)

[python-dev]

New changeset 8ec721bb3027 by Serhiy Storchaka in branch '3.5':
Issue bpo-26117: The os.scandir() iterator now closes file descriptor not only
https://hg.python.org/cpython/rev/8ec721bb3027

New changeset ec12fbf449a5 by Serhiy Storchaka in branch 'default':
Issue bpo-26117: The os.scandir() iterator now closes file descriptor not only
https://hg.python.org/cpython/rev/ec12fbf449a5

[vstinner]

Thanks for the fix Serhiy ;-)