Lazy imports

[Kronuz]

Lazy Imports

This implements Lazy Imports (PEP 690) on CPython main branch.

Most significant changes are in the following files:

Python/import.c - ~800 changes
Objects/dictobject.c - 600 changes
Python/ceval.c - ~300 changes
Objects/lazyimportobject.c - ~200 new lines

---

Python/import.c

• Implements the lazy imports basic functions for resolving objects and creating PyLazyImportObject objects.
• Implements: _PyImport_LazyImportName(), _PyImport_EagerImportName(), _PyImport_ImportFrom(), PyImport_LoadLazyImport() and _imp._maybe_set_submodule_attribute()
• Adds the C API and Python (_imp) API: PyImport_SetLazyImports(), PyImport_SetLazyImportsInModule(), PyImport_IsLazyImportsEnabled()

---

Objects/dictobject.c

• Makes necessary changes to dictionary internals to support lazy values.
• Resolved objects from a dictionary immediately and automatically replace the lazy objects that represented them in the dictionary (as long as it's possible). In the case where the value in the dictionary can't be immediately replaced by the resolved object (because maybe the resolution of the object triggered a dictionary mutation), the next time the lazy value it's accessed it'll use an internal cached version of the resolved object and attempt substituting the lazy value with the resolved one again.
• Values in a dictionary are resolved only when is needed, otherwise they remain lazy for as long as possible.
  • Merging, copying, dict.keys(), etc. all maintain lazy values inside a dictionary.
• _Py_dict_lookup() can now return DKIX_VALUE_ERROR in case the resolution of a lazy object resulted in an error.
• Adds *_keep_lazy/*KeepLazy (e.g. _Py_dict_lookup_keep_lazy(), _PyDict_GetItemKeepLazy()) to force returning lazy objects without being resolved.
• Adds PyDict_NextWithError(), which works the same way as PyDict_Next() with the exception it propagates any errors to the caller by returning 0 and setting an exception. Caller should use if (PyErr_Ocurred()) to check for any errors.
• Implements _PyDict_HasLazyImports()
• PyDict_Next() and PyDict_NextWithError() resolve all the lazy objects in the dictionary if it has them and if the passed pos is zero. Returns 0 on errors or if objects can't be resolved.
• Implements PyDict_ResolveLazyImports() to resolve all lazy values in a dictionary.
  • If there are no lazy objects in a dictionary it does nothing, otherwise it walks the whole dictionary trying to resolve each value and retrying if the dictionary mutated halfway.
  • It's used by dictionary views, PyDict_Next(), dict.values(), dict.items() before starting to iterate through the dictionary, to ensure returned values are all resolved and that the dictionary doesn't mutate midway due to import side effects produced by resolving the values.
  • The primary reason we need dk_lazy_imports is to be able to make this call efficient for dictionaries without lazy values.
• Implements PyDict_IsLazyImport() to check if a given key in a dictionary is a lazy object.

---

Python/ceval.c

• Implements EAGER_IMPORT_NAME and IMPORT_NAME to create PyLazyImportObject objects when the feature is enabled.
• It also resolves some lazy import objects on specialized opcodes.
• import_name() and import_from() were moved to Python/import.c, and mostly only renamed to _PyImport_EagerImportName() and _PyImport_ImportFrom(), respectively.

---

Objects/lazyimportobject.c

• Adds the implementation for PyLazyImportObject object via PyLazyImport_Type.
• Adds _PyLazyImport_NewModule(), _PyLazyImport_NewObject(), _PyLazyImport_GetName(), PyLazyImport_CheckExact()

[encukou]

I don't think the new API should be in the Stable ABI, at least initially. The few (hopefully) modules that need these can import and call importlib functions.
Except for PyDict_NextWithError -- that's an improvement, independent of lazy imports. In fact, if you add it now I'd be happy to review the PR.

Also: new items are generally added at the end of stable_abi.toml.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[Kronuz]

Rebased on top of CPython 3.12 stable.