Add _Py_PreInitialize() function

[vstinner]

BPO	36301
Nosy	@vstinner
PRs	bpo-36301: Add _PyWstrList structure #12343 bpo-36301: _PyCoreConfig_Read() ensures that argv is non-empty #12347 bpo-36301: Add _PyRuntimeState.preconfig #12350 [WIP] bpo-36301: Add _Py_PreInitialize() function #12351 bpo-36301: Fix Py_Main() memory leaks #12420 bpo-36301: Error if decoding pybuilddir.txt fails #12422 bpo-36301: Add _PyRuntime.pre_initialized #12457 bpo-36301: Add _PyPreCmdline internal API #12458 bpo-36301: Add _PyRuntimeState.preconfig #12506 bpo-36301: Cleanup preconfig code #12535 bpo-36301: Add _Py_PreInitializeFromConfig() #12536 bpo-36301: test_embed separates pre_config #12540 bpo-36301: Add _Py_GetEnv() function #12542 bpo-36301: Remove _PyCoreConfig.preconfig #12546 bpo-36301: Cleanup preconfig.c and coreconfig.c #12563 bpo-36301: Test Python init with isolated #12569 bpo-36301: Fix _PyPreConfig_Read() compiler warning #12695

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-26.23:55:16.532>
created_at = <Date 2019-03-15.12:29:28.556>
labels = ['interpreter-core', '3.8']
title = 'Add _Py_PreInitialize() function'
updated_at = <Date 2019-04-05.09:44:09.603>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-04-05.09:44:09.603>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-26.23:55:16.532>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-03-15.12:29:28.556>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36301
keywords = ['patch']
message_count = 20.0
messages = ['337982', '337991', '337993', '338287', '338303', '338428', '338440', '338484', '338664', '338809', '338810', '338814', '338829', '338837', '338846', '338895', '338903', '338923', '338925', '339483']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['12343', '12347', '12350', '12351', '12420', '12422', '12457', '12458', '12506', '12535', '12536', '12540', '12542', '12546', '12563', '12569', '12695']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36301'
versions = ['Python 3.8']

[vstinner]

Follow-up of bpo-36142, add _Py_PreInitialize() function to "pre-initialize" Python:

• initialize memory allocators
• initialize LC_CTYPE locale and UTF-8 Mode

Py_Initialize() should also be modified to no longer coerce the C locale or enable the UTF-8 Mode:
https://bugs.python.org/issue36202#msg337915

See also:

• bpo-36202: Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake
• bpo-36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?

[vstinner]

New changeset 74f6568 by Victor Stinner in branch 'master':
bpo-36301: Add _PyWstrList structure (GH-12343)
74f6568

[vstinner]

New changeset 6259976 by Victor Stinner in branch 'master':
bpo-36301: _PyCoreConfig_Read() ensures that argv is not empty (GH-12347)
6259976

[vstinner]

New changeset c183444 by Victor Stinner in branch 'master':
bpo-36301: Fix Py_Main() memory leaks (GH-12420)
c183444

[vstinner]

New changeset 5f9cf23 by Victor Stinner in branch 'master':
bpo-36301: Error if decoding pybuilddir.txt fails (GH-12422)
5f9cf23

[vstinner]

New changeset f29084d by Victor Stinner in branch 'master':
bpo-36301: Add _PyRuntime.pre_initialized (GH-12457)
f29084d

[vstinner]

New changeset fa15376 by Victor Stinner in branch 'master':
bpo-36301: Add _PyPreCmdline internal API (GH-12458)
fa15376

[vstinner]

In term of API, we get something like:

_PyInitError err;

_PyPreConfig preconfig = _PyPreConfig_INIT;
preconfig.utf8_mode = 1;
preconfig.allocator = "malloc";

    _PyInitError err = _Py_PreInitializeFromPreConfig(&preconfig);
    if (_Py_INIT_FAILED(err)) {
        _Py_ExitInitError(err);
    }

/* PyMem_RawMalloc and Py_DecodeLocale can now be used */

_PyCoreConfig config = _PyCoreConfig_INIT;
config.user_site_directory = 0;

    err = _Py_InitializeFromConfig(&config);
    if (_Py_INIT_FAILED(err)) {
        _Py_ExitInitError(err);
    }

/* ... use Python ... */

    Py_Finalize();

/* Note: no need to "free" preconfig nor config memory, they use constants */

[vstinner]

New changeset 6d5ee97 by Victor Stinner in branch 'master':
bpo-36301: Add _PyRuntimeState.preconfig (GH-12506)
6d5ee97

[vstinner]

New changeset f72346c by Victor Stinner in branch 'master':
bpo-36301: Cleanup preconfig code (GH-12535)
f72346c

[vstinner]

Note for myself: is there a problem between the priority of PYTHONHOME env var and pybuilddir.txt configuration file?

[vstinner]

New changeset a6fbc4e by Victor Stinner in branch 'master':
bpo-36301: Add _Py_PreInitializeFromConfig() (GH-12536)
a6fbc4e

[vstinner]

New changeset 1075d16 by Victor Stinner in branch 'master':
bpo-36301: Add _Py_GetConfigsAsDict() function (GH-12540)
1075d16