Useful (expensive) information is discarded in getpath.c.

[ericsnowcurrently]

BPO	45211
Nosy	@gvanrossum, @vstinner, @ericsnowcurrently, @corona10
PRs	bpo-45211: Move helpers from getpath.c to internal API. #28550 bpo-45211: Fix the Windows buildbots. #28584 bpo-45211: Add _Py_IsStdlibDir(). #28585 bpo-45211: Remember the stdlib dir during startup. #28586 bpo-45471: Do not set _Py_path_config.stdlib_dir in Py_SetPythonHome(). #28954

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-29.14:14:04.329>
created_at = <Date 2021-09-15.17:19:45.749>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Useful (expensive) information is discarded in getpath.c.'
updated_at = <Date 2021-10-14.20:48:44.350>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2021-10-14.20:48:44.350>
actor = 'eric.snow'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-29.14:14:04.329>
closer = 'eric.snow'
components = ['Interpreter Core']
creation = <Date 2021-09-15.17:19:45.749>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 45211
keywords = ['patch']
message_count = 10.0
messages = ['401860', '401867', '401900', '401943', '402730', '402798', '402814', '402815', '402876', '403944']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'vstinner', 'eric.snow', 'corona10']
pr_nums = ['28550', '28584', '28585', '28586', '28954']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45211'
versions = ['Python 3.11']

[ericsnowcurrently]

Currently we calculate a number of filesystem paths during runtime initialization in Modules/getpath.c (with the key goal of producing what will end up in sys.path). Some of those paths are preserved and some are not. In cases where the discarded data comes from filesystem access, we should preserve as much as possible.

The most notable info is location of the stdlib source files. We would store this as PyConfig.stdlib_dir (and _PyPathConfig.stdlib_dir). We'd expose it with sys.stdlibdir (or sys.get_stdlib_dir() if we might need to calculate lazily), similar to sys.platlibdir, sys.home, and sys.prefix.

sys.stdlibdir would allow us to avoid filesystem access, for example:

• in site.py
• in sysconfig.py
• detect if python is running out of the source tree (needed for bpo-45020)

FYI, I have a branch that mostly does what I'm suggesting here.

[gvanrossum]

Honestly I find it debatable whether we're doing anyone a favor by publishing the __file__ of the corresponding stdlib file for frozen modules. There will be situations where this points to the wrong file, and editing the file will not have an effect (unless you rebuild). I'd rather tell people to use -X frozen_modules=off until they can fix their dependency on the __file__ of frozen modules.

[ericsnowcurrently]

Setting __file__ on frozen modules is only one example. I actually need the stdlib dir to be preserved for other uses.

FWIW, I think you're probably right about __file__ on frozen modules. That said, further discussion about __file__ (or __path__) is probably more suitable in bpo-21736.

[vstinner]

See also #23169 of bpo-42260 which rewrites getpath.c in pure Python.

[ericsnowcurrently]

New changeset ae7839b by Eric Snow in branch 'main':
bpo-45211: Move helpers from getpath.c to internal API. (gh-28550)
ae7839b

[ericsnowcurrently]

New changeset 0c50b8c by Eric Snow in branch 'main':
bpo-45211: Remember the stdlib dir during startup. (gh-28586)
0c50b8c

[ericsnowcurrently]

I have what I need for now (stdlib dir). There may be more info to preserve, but I'll leave it others to pursue that.