os.closerange optimization

[ferringb]

BPO	13788
Nosy	@birkenfeld, @gpshead, @ronaldoussoren, @vstinner, @tiran, @wmanley
Files	closerange-optimization.patch: for non-windows systems, if /proc/${PID}/fd is available make use of it

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-01-15.04:29:24.624>
labels = ['extension-modules', '3.11', 'performance']
title = 'os.closerange optimization'
updated_at = <Date 2021-07-05.00:19:38.896>
user = 'https://bugs.python.org/ferringb'

bugs.python.org fields:

activity = <Date 2021-07-05.00:19:38.896>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2012-01-15.04:29:24.624>
creator = 'ferringb'
dependencies = []
files = ['24242']
hgrepos = []
issue_num = 13788
keywords = ['patch']
message_count = 12.0
messages = ['151273', '151274', '151275', '151287', '151289', '192738', '192739', '192760', '192776', '351354', '396819', '396963']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'gregory.p.smith', 'ronaldoussoren', 'vstinner', 'ferringb', 'christian.heimes', 'rosslagerwall', 'wmanley']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue13788'
versions = ['Python 3.11']

[ferringb]

The current implementation of closerange essentially is a bruteforce invocation of close for every integer in the range.

While this works, it's rather noisy for stracing, and for most invocations, is near a thousand close invocations more than needed.

As such it should be aware of /proc/${PID}/fd, and use that to isolate down just what is actually open, and close that.

[ferringb]

Fixed tabs/spaces...

[rosslagerwall]

Thanks for the patch.

However, this cannot as far as I understand be used for the subprocess implementation due to the limitation of what can be called after a fork() and before an exec().

Take a look at bpo-8052 for some more discussion of this.

[birkenfeld]

fwiw, s/MSDOS_WINDOWS/MS_WINDOWS/.

[gpshead]

Reopening. Comments added to the code review.

This issue is independent of the subprocess module issue in bpo-8052. The _posixsubprocess.c has its own fd closing loop.

http://hg.python.org/cpython/file/050c07b31192/Modules/_posixsubprocess.c#l118

[ronaldoussoren]

Two small technical comments:

1. I'd add a configure or compile-time check to determine if the procfs
   interface might be available. I don't like probing for features that
   you know are not available.

2. MacOSX has simular functionality using /dev/fd instead of
   /proc/${PID}/fd (and other BSD systems might have this as well)

[tiran]

In case someone is wondering if the approach really reduces the amount of syscalls: yes, it does. readdir() doesn't do a syscall for each entry. On Linux it uses the internal syscall getdents() to fill a buffer of directory entry structs. http://man7.org/linux/man-pages/man2/getdents.2.html

On my system os.listdir() does four syscalls:

$ strace python -c "import os; os.listdir('/home/heimes')"

openat(AT_FDCWD, "/home/heimes", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, /* 381 entries */, 32768) = 12880
getdents(3, /* 0 entries */, 32768) = 0
close(3)

On Linux you can also use /proc/self/fd instead of /proc/YOURPID/fd.

Other operating systems have different APIs to get a list of open FDs. AFAK /dev/fd is static on FreeBSD and Mac OS X:

FreeBSD:
http://www.manualpages.de/FreeBSD/FreeBSD-7.4-RELEASE/man3/kinfo_getfile.3.html

Darwin / Mac OS X:
proc_pidinfo()

[gpshead]

_posixsubprocess already uses the Linux getdent64 syscall when available (though for different reasons: readdir is not safe in that context). http://hg.python.org/cpython/file/3f3cbfd52f94/Modules/_posixsubprocess.c#l227

Probing for procfs at configure time could be problematic. It is a virtual filesystem. It is entirely possible for a system to choose not to mount it. It might be reasonable to assume that it "might be present" only if the system had it mounted at compile time but a configure flag to override that might be desirable for some systems (not the Linux systems I usually deal with).

If we're going through all of these hoops for closerange: I'd love to see an API exposed in the os module to return a list of open fd's. It is an abstraction nobody should have to write for themselves.

[vstinner]

FreeBSD and other OSes provide closefrom(). Why not exposing this function which is probably implemented as a single syscall?