[C API] Add _PyInterpreterState_SetConfig(): reconfigure an interpreter

[vstinner]

BPO	42260
Nosy	@ncoghlan, @vstinner, @serhiy-storchaka, @zooba, @cmeyer, @shihai1991
PRs	bpo-42260: Reorganize PyConfig #23149 bpo-42260: Main init modify sys.flags in-place #23150 bpo-42260: Add _PyInterpreterState_SetConfig() #23158 bpo-42260: Add _PyConfig_FromDict() #23167 bpo-42260: PyConfig_Read() only parses argv once #23168 [WIP] bpo-42260: Rewrite getpath.c in Python #23169 bpo-42260: Compute the path config in the main init #23211 bpo-42260: Fix _PyConfig_Read() if compute_path_config=0 #23220 bpo-42260: Initialize warnings and time early at startup #23249 bpo-42260: Improve error handling in _PyConfig_FromDict #23488

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-21.21:56:32.186>
created_at = <Date 2020-11-04.14:45:21.168>
labels = ['expert-C-API', '3.10']
title = '[C API] Add _PyInterpreterState_SetConfig(): reconfigure an interpreter'
updated_at = <Date 2021-10-26.21:11:34.282>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-10-26.21:11:34.282>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-21.21:56:32.186>
closer = 'vstinner'
components = ['C API']
creation = <Date 2020-11-04.14:45:21.168>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42260
keywords = ['patch']
message_count = 20.0
messages = ['380327', '380328', '380341', '380382', '380421', '380424', '380657', '380658', '380659', '380707', '380728', '380773', '380775', '380780', '380781', '380823', '381711', '381718', '381782', '402376']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'vstinner', 'serhiy.storchaka', 'steve.dower', 'cmeyer', 'shihai1991']
pr_nums = ['23149', '23150', '23158', '23167', '23168', '23169', '23211', '23220', '23249', '23488']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42260'
versions = ['Python 3.10']

[vstinner]

This issue is a follow-up of the PEP-567 which introduced the PyConfig C API and is related to PEP-432 which wants to rewrite Modules/getpath.c in Python.

I would like to add a new PyInterpreterState_SetConfig() function to be able to reconfigure a Python interpreter in C. One example is to write a custom sys.path, to implement of virtual environment (common request for embedded Python), etc. Currently, it's really complex to tune the Python configuration.

The use case is to tune Python for embedded Python. First, I would like to add new functions to the C API for that:

• PyInterpreterState_GetConfigCopy()
• PyInterpreterState_SetConfig()

The second step will to be expose these two functions in Python (I'm not sure where for now), and gives the ablity to tune the Python configuration in pure Python.

The site module already does that for sys.path, but it is running "too late" in the Python initialization. Here the idea is to configure Python before it does access any file on disk, after the "core" initialization and before the "main" initialization.

One concrete example would be to reimplement Modules/getpath.c in Python, convert it to a frozen module, and run it at Python startup to populate sys.path. It would allow to move some of the site code into this module to run it earlier.

Pseudo-code in C:
---------------------

void init_core(void)
{
  // "Core" initialization
  PyConfig config;
  PyConfig_InitPython(&config);
  PyConfig._init_main = 0
  Py_InitializeFromc(&config);
  PyConfig_Clear(&config);
}

void tune_config(void)
{
  PyConfig config;
  PyConfig_InitPython(&config);

// Get a copy of the current configuration
PyInterpreterState_GetConfigCopy(&config); // <== NEW API!

// ... put your code to tune config ...

// dummy example, current not possible in Python
config.bytes_warnings = 1;

  // Reconfigure Python with the updated configuration
  PyInterpreterState_SetConfig(&config);  // <=== NEW API!
  PyConfig_Clear(&config);
}
  
int main()
{
  init_core();
  tune_config(); // <=== THE USE CASE!
  _Py_InitializeMain();
  return Py_RunMain();
}

---

In this example, tune_config() is implemented in C. But later, it will be possible to convert the configuration to a Python dict and run Python code to tune the configuration.

The PEP-587 added a "Multi-Phase Initialization Private Provisional API":

• PyConfig._init_main = 0
• _Py_InitializeMain()

https://docs.python.org/dev/c-api/init_config.html#multi-phase-initialization-private-provisional-api

[vstinner]

New changeset cfb41e8 by Victor Stinner in branch 'master':
bpo-42260: Reorganize PyConfig (GH-23149)
cfb41e8

[vstinner]

New changeset af1d64d by Victor Stinner in branch 'master':
bpo-42260: Main init modify sys.flags in-place (GH-23150)
af1d64d

[vstinner]

New changeset 048a356 by Victor Stinner in branch 'master':
bpo-42260: Add _PyInterpreterState_SetConfig() (GH-23158)
048a356

[vstinner]

New changeset f3cb814 by Victor Stinner in branch 'master':
bpo-42260: Add _PyConfig_FromDict() (GH-23167)
f3cb814

[vstinner]

New changeset dc42af8 by Victor Stinner in branch 'master':
bpo-42260: PyConfig_Read() only parses argv once (GH-23168)
dc42af8

[vstinner]

New changeset 9e1b828 by Victor Stinner in branch 'master':
bpo-42260: Compute the path config in the main init (GH-23211)
9e1b828

[vstinner]

If we remove Modules/getpath.c, it will no longer be possible to automatically computes the path configuration when one of the following getter function will be called:

• Py_GetPath()
• Py_GetPrefix()
• Py_GetExecPrefix()
• Py_GetProgramFullPath()
• Py_GetPythonHome()
• Py_GetProgramName()

It means that these functions would not return NULL if called before Python is initialiazed, but return the expected string once Python is initialized.

Moreover, Py_SetPath() would no longer automatically computes the "program full path" (sys.executable).

[vstinner]

If we remove Modules/getpath.c, it will no longer be possible to automatically computes the path configuration when one of the following getter function will be called: (...)

It is not really an incompatible change according to the documentation:

"Note: The following functions should not be called before Py_Initialize(): Py_EncodeLocale(), Py_GetPath(), Py_GetPrefix(), Py_GetExecPrefix(), Py_GetProgramFullPath(), Py_GetPythonHome(), Py_GetProgramName() and PyEval_InitThreads().".

https://docs.python.org/dev/c-api/init.html

[vstinner]

New changeset ace3f9a by Victor Stinner in branch 'master':
bpo-42260: Fix _PyConfig_Read() if compute_path_config=0 (GH-23220)
ace3f9a

[vstinner]

The main drawback of rewriting Modules/getpath.c as Lib/_getpath.py (and removing getpath.c) is that PyConfig_Read() could no longer compute the Python Path Configuration. It would return an "empty" path configuration.