Add stat caching option to pathlib

[gvanrossum]

BPO	26031
Nosy	@gvanrossum, @pitrou
Files	statcache.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-04.21:43:41.496>
created_at = <Date 2016-01-06.22:41:29.808>
labels = ['type-feature']
title = 'Add stat caching option to pathlib'
updated_at = <Date 2020-11-04.21:43:41.495>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2020-11-04.21:43:41.495>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-04.21:43:41.496>
closer = 'gvanrossum'
components = []
creation = <Date 2016-01-06.22:41:29.808>
creator = 'gvanrossum'
dependencies = []
files = ['41521']
hgrepos = []
issue_num = 26031
keywords = ['patch']
message_count = 4.0
messages = ['257651', '257652', '257658', '380366']
nosy_count = 2.0
nosy_names = ['gvanrossum', 'pitrou']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26031'
versions = ['Python 3.6']

[gvanrossum]

There are concerns that pathlib is inefficient because it doesn't cache stat() operations. Thus, for example this code calls stat() for each result twice (once internal to the glob, a second time to answer the is_symlink() question):

  p = pathlib.Path('/usr')
  links = [x for x in p.rglob('*') if x.is_symlink()]

I have a tentative patch (without tests). On my Mac it only gives modest speedups (between 5 and 20 percent) but things may be different on other platforms or for applications that make a lot of inquiries about the same path.

The API I am proposing is that by default nothing changes; to benefit from caching you must instantiate a StatCache() object and pass it to Path() constructor calls, e.g. Path('/usr', stat_cache=cache_object). All Path objects derived from this path object will share the cache. To force an uncached Path object you can use Path(p).

The patch is incomplete; there are no tests for the new functionality (though existing tests pass) and __eq__ should be adjusted so that Path objects using different caches always compare unequal.

Question for Antoine: Did you perhaps anticipate a design like this? Each Path instance has an _accessor slot, but there is only one accessor instance defined that is used everywhere (the global _normal_accessor). So you could have avoided a bunch of complexity in the code around setting the proper _accessor unless you were planning to use multiple accessors.

[pitrou]

Let me first mention the stat caching question was asked during the PEP discussion and ultimately it was decided it's not a good idea to meld it in the Path design :-)

Early versions of pathlib were more complex as they were able to keep some file descriptors around, for example for openat() support. You can probably find them by digging in the original repo.

[gvanrossum]

That's fair, though I don't know what kind of caching design was considered
(until Ram's suggestion on python-ideas I had thought the cache would
simply use a slot on the Path instance to hold the stat() result).

If we want pathlib to become pervasive we may have to accept that its
implementation may become more complex in order to support more use cases.
(I'm so far holding off a walk() method, but who knows for how long...)

[gvanrossum]

Okay, I am giving up on this.