Py_DecodeLocale() fails if used before the runtime is initialized.

[ericsnowcurrently]

BPO	32096
Nosy	@ncoghlan, @vstinner, @ericsnowcurrently, @serhiy-storchaka, @AraHaan
PRs	bpo-32096: Statically initialize the default raw allocator. #4481 bpo-32096: Fall back to defaults in PyMem_RawMalloc() and PyMem_RawFree(). #4495 bpo-32096: Remove obj and mem from _PyRuntime #4532 bpo-32096: Ensure test can find the encodings module #4566

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2017-11-26.05:41:53.556>
created_at = <Date 2017-11-20.21:28:36.310>
labels = ['interpreter-core', '3.7', 'type-crash']
title = 'Py_DecodeLocale() fails if used before the runtime is initialized.'
updated_at = <Date 2017-11-26.05:41:53.555>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2017-11-26.05:41:53.555>
actor = 'ncoghlan'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2017-11-26.05:41:53.556>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2017-11-20.21:28:36.310>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 32096
keywords = ['patch']
message_count = 32.0
messages = ['306585', '306587', '306591', '306614', '306654', '306655', '306656', '306658', '306659', '306660', '306662', '306689', '306708', '306717', '306718', '306737', '306740', '306764', '306766', '306769', '306772', '306774', '306889', '306893', '306895', '306896', '306938', '306949', '306982', '306987', '306989', '306993']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'vstinner', 'eric.snow', 'serhiy.storchaka', 'Decorater']
pr_nums = ['4481', '4495', '4532', '4566']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue32096'
versions = ['Python 3.7']

[ericsnowcurrently]

(see the python-dev thread [1])
(related: issue bpo-32086)

When I consolidated the global runtime state into a single global, _PyRuntime, calls Py_DecodeLocale() started to break if the runtime hadn't been intialized yet. This is because that function relies on PyMem_RawMalloc() and PyMem_RawFree(), which rely on the raw allocator having been initialized as part of the runtime (it used to be intialized statically).

The documentation for various "Process-wide parameters" [2] explicitly directs users to call Py_DecodeLocale() where necessary. The docs for Py_DecodeLocale(), in turn, explicitly refer to calling PyMem_RawFree(). So changing the pre-runtime-init behavior of Py_DecodeLocale() and PyMem_RawFree() is a regression that should be fixed.

[1] https://mail.python.org/pipermail/python-dev/2017-November/150605.html
[2] https://docs.python.org/3/c-api/init.html#process-wide-parameters

[AraHaan]

Interesting, on 3.6.3 on my embedded program it seems to work just fine.

Did anything change in it since then?

https://github.com/AraHaan/Els_kom_new/blob/master/PC/komextract_new.c

[vstinner]

Duplicate of bpo-32086.

[ncoghlan]

While they're definitely closely related, I don't think this and bpo-32086 are actually duplicates: this issue is "fix the current Py_DecodeLocale regression in 3.7 by reverting back to the 3.6 behaviour", while bpo-32086 is "Should we deprecate our implied support for calling Py_DecodeLocale() before calling Py_Initialize()?".

The latter is certainly a valid question, but we should restore the 3.6 behaviour while we're considering it.

[ericsnowcurrently]

I see at least 3 ways to sort this out:

1. partially revert the _PyRuntime change, sort of temporarily
   ("revert the change on memory allocators, and retry later to fix
   it, once other initializations issues are fixed", as suggested by
   Victor in the email thread)
2. statically initialize the "raw" allocator with defaults, enough
   to make PyMem_RawMalloc() and PyMem_RawFree() work pre-init (this
   is what my PR does)
3. use hard-coded defaults in PyMem_RawMalloc() and PyMem_RawFree()
   if the runtime has not been initialized yet

I considered implementing #3 instead, but wasn't sure about the performance impact. It would add a pointer comparison to NULL and a branch on each PyMem_RawMalloc() and PyMem_RawFree() call. I'm not convinved it would make much of a difference. Furthermore, I consider #3 to be the simplest solution, both to implement and to maintain, so I'll probably try it out.

[ericsnowcurrently]

I thought issue bpo-32086 was about documentation (which is worth having a separate issue for), not about a fix to the regression.

[vstinner]

"3. use hard-coded defaults in PyMem_RawMalloc() and PyMem_RawFree() if the runtime has not been initialized yet"

I dislike this option since it can have a negative impact on performances. The PEP-445 already added a new level of indirection and so made memory allocations a little bit slower.

[vstinner]

I marked bpo-32096 as a duplicate of this one. I don't want to discuss the same issue in 3 places (2 bpo and python-dev).

[vstinner]

Nick: "Should we deprecate our implied support for calling Py_DecodeLocale() before calling Py_Initialize()?"

Please don't do that. Py_DecodeLocale() is the best available function to decode paths and environment variables to initialize Python. The implementation of Py_DecodeLocale() is very complex, but we need this complexity to decode "correctly" OS data.