Expose _PyCoreConfig structure to Python

[warsaw]

BPO	33919
Nosy	@warsaw, @ncoghlan, @vstinner, @tiran, @ericsnowcurrently, @emilyemorehouse
PRs	bpo-33919: Expose hash_seed and use_hash_seed in sys.hash_info #7867

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/warsaw'
closed_at = <Date 2019-05-27.23:40:30.967>
created_at = <Date 2018-06-20.20:06:15.391>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'Expose _PyCoreConfig structure to Python'
updated_at = <Date 2019-05-27.23:40:30.966>
user = 'https://github.com/warsaw'

bugs.python.org fields:

activity = <Date 2019-05-27.23:40:30.966>
actor = 'vstinner'
assignee = 'barry'
closed = True
closed_date = <Date 2019-05-27.23:40:30.967>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-06-20.20:06:15.391>
creator = 'barry'
dependencies = []
files = []
hgrepos = []
issue_num = 33919
keywords = ['patch']
message_count = 19.0
messages = ['320107', '320108', '320109', '320111', '320198', '320211', '320250', '320265', '320267', '320269', '320270', '320285', '320289', '320299', '320345', '320347', '320438', '342530', '343706']
nosy_count = 6.0
nosy_names = ['barry', 'ncoghlan', 'vstinner', 'christian.heimes', 'eric.snow', 'emilyemorehouse']
pr_nums = ['7867']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33919'
versions = ['Python 3.8']

[warsaw]

The _PyCoreConfig structure in pystate.h has some interesting fields that I don't think are exposed anywhere else to Python-land. I was particularly interested recently in hash_seed and use_hash_seed. I'm thinking that it may be useful to expose this structure in the sys module.

[tiran]

hash_seed and use_hash_seed could be added to sys.hash_info. This would be the first place I'd look for the information. After all I implemented it. :)

[warsaw]

On Jun 20, 2018, at 13:28, Christian Heimes <report@bugs.python.org> wrote:

Christian Heimes <lists@cheimes.de> added the comment:

hash_seed and use_hash_seed could be added to sys.hash_info. This would be the first place I'd look for the information. After all I implemented it. :)

That was the first place I looked too :)

[vstinner]

Is is still a secret seed if it's public? :)

[warsaw]

I think the basic implementation problem is that by the time you get to get_hash_info() in sysmodule.c, you no longer have access to the _PyCoreConfig object, nor the _PyMain object that it's generally attached to.

[vstinner]

I think the basic implementation problem is that by the time you get to get_hash_info() in sysmodule.c, you no longer have access to the _PyCoreConfig object, nor the _PyMain object that it's generally attached to.

An interpreter now keeps a copy of _PyCoreConfig and _PyMainInterpreterConfig.

See for example make_flags() in sysmodule.c:

_PyCoreConfig *core_config = &_PyGILState_GetInterpreterStateUnsafe()->core_config;
...
PyStructSequence_SET_ITEM(seq, pos++, PyBool_FromLong(core_config->dev_mode));

The interpreter really owns the copy of these configs and they are kept until the interpreter object is destroyed.

Another example:

static PyObject *
import_find_and_load(PyObject *abs_name)
{
    ...
    int import_time = interp->core_config.import_time;
    ...
}

[warsaw]

Thanks for the hint! I had a feeling there had to be an API to get at it, but I couldn’t find it. Maybe we should start documenting the Python Secret Underscore API? :)

On Jun 22, 2018, at 00:04, STINNER Victor <report@bugs.python.org> wrote:

_PyCoreConfig *core_config = &_PyGILState_GetInterpreterStateUnsafe()->core_config;
...
PyStructSequence_SET_ITEM(seq, pos++, PyBool_FromLong(core_config->dev_mode));

The interpreter really owns the copy of these configs and they are kept until the interpreter object is destroyed.

[warsaw]

I think there's another thing I'd like to change, and it seems like it's "just" an implementation detail. In _Py_HashRandomization_Init(), if use_hash_seed is 0, then we directly inject the random bits into the buffer, and then there's no hash_seed. I'd like to change that so that if use_hash_seed is 0, then we create a random hash seed first, and then call lcg_urandom() for the hash secret. That way, even if Python itself uses a random hash seed, we'll have a record of that in the runtime that can be used to reproduce the hashing. In this case, I'd still leave use_hash_seed == 0, and that would tell you what combinations of env vars were used.

[vstinner]

Nick plans to finish his PEP-432 for Python 3.8 and make the API public.
See with him? The PEP should document these structures but I was ahead and
made changes which were not scheduled and the PEP is now outdated.

[warsaw]

Nosying Nick. I agree there's some overlap with Python startup restructuring, but it feels kind of orthogonal too. I really am only exposing (some elements) of that structure to Python.

What might be interesting though would be if we want to expose the entire structure and not just the hash seeds, as I'm leaning toward here (given that we already have sys.hash_info).

[vstinner]

Barry: generating a 32 bit seed gives less entropy and so makes Python
easier to crash. If you need reproducible Python: generate a seed and set
the env var before starting Python. Tox does that. Regrtest should do that.