Skip to content

GH-115060: Speed up pathlib.Path.glob() by not scanning literal parts - #117732

Merged
barneygale merged 4 commits into
python:mainfrom
barneygale:gh-117586-again-again
Apr 12, 2024
Merged

GH-115060: Speed up pathlib.Path.glob() by not scanning literal parts#117732
barneygale merged 4 commits into
python:mainfrom
barneygale:gh-117586-again-again

Conversation

@barneygale

@barneygale barneygale commented Apr 10, 2024

Copy link
Copy Markdown
Contributor

Don't bother calling os.scandir() to scan for literal pattern segments, like foo in foo/*.py. Instead, append the segment(s) as-is and call through to the next selector with exists=False, which signals that the path might not exist. Subsequent selectors will call os.scandir() or os.lstat() to filter out missing paths as needed.

Timings:

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib'))"
5000 loops, best of 5: 69.4 usec per loop
20000 loops, best of 5: 13.6 usec per loop
# --> 5.1x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/'))"
5000 loops, best of 5: 73.3 usec per loop
20000 loops, best of 5: 14.2 usec per loop
# --> 5.16x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*'))"
1000 loops, best of 5: 362 usec per loop
1000 loops, best of 5: 301 usec per loop
# --> 1.2x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*/__init__.py'))"
200 loops, best of 5: 1.18 msec per loop
1000 loops, best of 5: 273 usec per loop
# --> 4.32x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**/__init__.py'))"
50 loops, best of 5: 9.46 msec per loop
50 loops, best of 5: 5.72 msec per loop
# --> 1.65x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/pathlib/__init__.py'))"
1000 loops, best of 5: 210 usec per loop
20000 loops, best of 5: 14.9 usec per loop
# --> 14.1x faster

…al parts

Don't bother calling `os.scandir()` to scan for literal pattern segments,
like `foo` in `foo/*.py`. Instead, append the segment(s) as-is and call
through to the next selector with `exists=False`, which signals that the
path might not exist. Subsequent selectors will call `os.scandir()` or
`os.lstat()` to filter out missing paths as needed.
@barneygale
barneygale merged commit 0eb52f5 into python:main Apr 12, 2024
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
…al parts (python#117732)

Don't bother calling `os.scandir()` to scan for literal pattern segments,
like `foo` in `foo/*.py`. Instead, append the segment(s) as-is and call
through to the next selector with `exists=False`, which signals that the
path might not exist. Subsequent selectors will call `os.scandir()` or
`os.lstat()` to filter out missing paths as needed.
SirVer added a commit to SirVer/ultisnips that referenced this pull request Sep 6, 2026
`find_all_snippet_directories()` expands the wildcards Vim allows in `'runtimepath'` entries. Since the pathlib migration (c84bb5a) it did so with `Path(pth.anchor).glob(str(pth.relative_to(pth.anchor)))`, i.e. it globbed the whole path starting at `/`. 

Python 3.12's pathlib has no fast path for literal pattern components: every component goes through  _WildcardSelector`, which `scandir`s the parent directory and calls `is_dir()` on each entry. With the ~100 entries of a plugin-heavy runtimepath that lists `/`, `/home`, `/home/user`, ... once per entry (595 directory listings and 5763 stats in the report), and on WSL with symlinked network folders the first snippet expansion froze Vim for seconds. 

Neither 3.11 nor 3.13 have this issue. 3.11 pathlib resolved literal pattern segments through a single `stat`. Python 3.12 removed that path in python/cpython#102710. Python 3.13 brought a literal fast path back in the glob rewrite (python/cpython#117732).

We work around this by finding globbing in the snippet paths and only start globbing from the segment that has globbing characters. While strictly only needed for 3.12 it does not hurt on other Pythons. 

Fixes #1694.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance or resource usage topic-pathlib

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant