Add _PyPreConfig and rework _PyCoreConfig and _PyMainInterpreterConfig

[vstinner]

BPO	35266
Nosy	@ncoghlan, @vstinner, @ambv, @ericsnowcurrently
PRs	[WIP] bpo-35266: Add _PyPreConfig #10575

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-12-17.11:28:01.728>
created_at = <Date 2018-11-16.14:44:06.422>
labels = ['interpreter-core', '3.8']
title = 'Add _PyPreConfig and rework _PyCoreConfig and _PyMainInterpreterConfig'
updated_at = <Date 2018-12-17.11:28:01.727>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-12-17.11:28:01.727>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-17.11:28:01.728>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-11-16.14:44:06.422>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35266
keywords = ['patch']
message_count = 6.0
messages = ['330005', '330039', '330069', '330252', '330263', '331987']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'vstinner', 'lukasz.langa', 'eric.snow']
pr_nums = ['10575']
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35266'
versions = ['Python 3.8']

[vstinner]

The C code of Python initialization uses _PyCoreConfig and _PyMainInterpreterConfig which co-exist and more or less redundant. For example, both structures have "argv": wchar_** for _PyCoreConfig, PyObject* (list of str) for _PyMainInterpreterConfig.

I propose to put _PyCoreConfig inside _PyMainInterpreterConfig and move wchar_t* strings into a new _PyPreConfig structure.

The main idea is that _PyPreConfig and wchar_* type are only used for early Python initialization, but once Python is initialization, the configuration should be provided as Python objects.

I didn't check all configuration variables to see if it's doable or not, but I wrote a proof-of-concept pull request to show my idea with concrete (and working!) code.

See attached PR.

[ncoghlan]

I like where you're going with this, but would be willing to write an update to PEP-432 to sketch out in advance what you now think the end state is going to look like?

Merging the general structure of the draft PEP-432 implementation to make it possible to start migrating settings and see what's viable in practice has pretty much worked out as hoped, but we've diverged far enough from that structure now that we can't credibly claim to be working towards the current PEP draft as the new multi stage initialisation API.

Changing the proposal (and adding yourself as a co-author) is fine - that was the whole point of enabling initial development as a private API in the first place.

[vstinner]

I like where you're going with this, but would be willing to write an update to PEP-432 to sketch out in advance what you now think the end state is going to look like?

Sadly, I'm unable to design in advance what will be the final state.

Python initialization is a giant beast, full of traps, with many practical issues.

I'm moving slowly, step by step.

For example, this issue "only" move wchar_t* out of _PyCoreConfig, but Eric Snow told that me that he (or you, Nick, I don't recall) would prefer to not use "Unicode" during the very first initialization stage. wchar_t* is already Unicode. I'm unable to see yet how to have 3 stages:

1. no unicode
2. C structures, wchar_t*
3. Python objects

Currently, (1)+(2) is _PyCoreConfig and (3) is _PyMainInterpreterConfig.

I prefer to work directly on the code to make sure to have a working implementation, than working on paper but don't know if it's possible to implement it :-)

One issue is that it requires more steps, but from my point of view we better control the risk since it's possible to move back if we make a mistake in a small change.

[ncoghlan]

I didn't know what was possible when I wrote PEP-432 either - instead, I wrote down an initial concept for what I *wanted*, and then started exploring the code to find out the barriers to achieving that.

We know enough now to know that original design concept isn't technically feasible, but that's OK - the general idea was just to get to a point where the startup code is better tested, easier to maintain, and easier to control in an embedding application, and everything outside that is negotiable.

The problem with the purely bottom-up approach is that we may end up with something that's better tested and easier to maintain, but find out that it hasn't actually helped us get to a point where we can make the interpreter easier for embedding applications to manage.

As far as Unicode goes, it isn't Unicode as a concept that's problematic, it's specifically the CPython Unicode type: that needs hash randomisation configured, and that means we need to have already processed the input settings that can affect the hash seed. And unlike UTF-8 mode, where there's a comparatively limited set of strings to recreate with a different decoding step, there's no escape hatch to let you cleanly recreate all previously created string objects with a different basis for their hash.

[vstinner]

Hum, my split is incomplete. From a high level point of view, the initialization should be done in these steps:

1. select memory allocator, config made of C char* (bytes) and int types
2. select encodings, add wchar_t* (Unicode) strings to the config
3. compute the "path configuration" (used to initialize importlib and sys.path)
4. apply the config to Python

Step (3) should be optional. Currently, the path configuration can be set in _PyCoreConfig to avoid the need "calculate" it (operation which access the filesystem).

Technically, we may have to use wchar_t* in (1), since Windows uses wmain() which gets argv as wchar_t** (and environment variables as wchar_t* as well).

[vstinner]

When I looked again at this issue, I'm not sure how what should be done, what is the proper design, what should stay after Python initialization, etc. I prefer to abandon this change and maybe retry to write it later.

I have a more advanced version in this branch of my fork: https://github.com/vstinner/cpython/commits/pre_config_next