[meta] Freeze commonly used stdlib modules.

[ericsnowcurrently]

BPO	45661
Nosy	@gvanrossum, @tiran, @methane, @ericsnowcurrently, @FFY00, @kumaraditya303

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-28.19:39:40.296>
labels = ['type-bug', 'build', '3.11']
title = '[meta] Freeze commonly used stdlib modules.'
updated_at = <Date 2022-01-07.20:05:59.914>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2022-01-07.20:05:59.914>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-10-28.19:39:40.296>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 45661
keywords = []
message_count = 7.0
messages = ['405239', '405240', '409947', '409948', '409958', '409959', '410024']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'christian.heimes', 'methane', 'eric.snow', 'FFY00', 'kumaraditya']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45661'
versions = ['Python 3.11']

[ericsnowcurrently]

We're already freezing modules needed for bootstrapping the runtime, as well as the modules imported during startup. [1][2] We're also planning on freezing argparse. [3] There may be other modules (or files) worth freezing (assuming we don't just freeze the entire stdlib, which would have some potential downsides).

This issue is meant to cover the broader idea. The work to actually freeze modules should be handled in separate issues.

[1] https://bugs.python.org/issue45020
[2] https://bugs.python.org/issue45654
[3] https://bugs.python.org/issue45660

[ericsnowcurrently]

One to consider that isn't necessarily used that commonly is sysconfig. However, that could also involve freezing the "sysconfigdata" file (without needing to expose it as a module).

[kumaraditya303]

functools and contextlib are very common so they should be deep-frozen. warnings and re would be nice to do.

[methane]

I don't against deep freezing functools and contextlib.
But I think we should optimize and utilize zipimport or something similar, because we can not deep-freeze all stdlib or 3rd party libraries.

See also: faster-cpython/ideas#158 (reply in thread)

[tiran]

Deep freezing is not a miracle cure for startup performance issues. Every additional frozen module increases the size of the main binary / shared library. This can have negative affects for performance and usability, too. For WASM builds we want to make the wasm binary as small as possible.

[kumaraditya303]

Can you elaborate a bit about "This can have negative affects for performance" as I think freezing modules has only one downside of increasing binary size ?

For WASM builds we want to make the wasm binary as small as possible.

If WASM would become an officially supported platform then there can be a patch to disable freezing on such platforms otherwise I don't see how because of WASM we cannot deepfreeze for other platforms which benefit from it.

[gvanrossum]

Deep-freezing is definitely not a miracle cure. We should continue to trim or delay unneeded imports (like Christian just did for setuptools' _distutils_hack).

What *would* be a miracle cure would be if we could deep-freeze the module contents after it has *executed*. But that is much harder in general, since two executions (even on the same platform) could result in different module contents (e.g. conditionally defining something based on environment contents).

Instagram's Cinder has something that works for this, strict modules (https://github.com/facebookincubator/cinder#strict-modules). It is a complex system! But if we could do this it could really speed up a lot of imports tremendously. (Though still at the cost of binary size increase.)