Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake #80383
Description
| BPO | 36202 |
|---|---|
| Nosy | @ncoghlan, @vstinner, @methane |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-03-27.17:29:58.167>
created_at = <Date 2019-03-06.00:53:52.873>
labels = ['interpreter-core', '3.8']
title = 'Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake'
updated_at = <Date 2019-03-27.17:29:58.166>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2019-03-27.17:29:58.166>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-27.17:29:58.167>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-03-06.00:53:52.873>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36202
keywords = ['patch']
message_count = 7.0
messages = ['337252', '337260', '337261', '337281', '337915', '338977', '338979']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'vstinner', 'methane']
pr_nums = ['12589']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36202'
versions = ['Python 3.8']
Activity
vstinner commentedon Mar 6, 2019
Calling Py_DecodeLocale() before Py_Initialize() or _Py_InitializeCore() is broken since Python 3.7 if the C locale coercion (PEP-538) or UTF-8 mode (PEP-540) changes the encoding in the middle of _Py_InitializeCore().
I added a new phase to the Python initialization in bpo-36142, a new _PyPreConfig structure, which can be used to fix this mojibake issue.
The code for embedding Python should look like:
---
_Py_PreInitialize();
_PyCoreConfig config;
config.home = Py_DecodeLocale("/path/to/home");
/* use Python here */
Except that there is no _Py_PreInitialize() function yet :-)
vstinner commentedon Mar 6, 2019
The "vim" editor embeds Python. It sets the Python home by calling Py_SetPythonHome() with the following code:
---
ref: https://github.com/vim/vim/blob/14816ad6e58336773443f5ee2e4aa9e384af65d2/src/if_python3.c#L874-L887
mbstowcs() uses the current LC_CTYPE locale. Python can select a different filesystem encoding than the LC_CTYPE encoding depending on PEP-538 and PEP-540. So encoding back the Python home to bytes to access to files on the filesystem can fail because of mojibake.
The code should by written like (pseudo-code):
---
_Py_PreInitialize();
The vim case has been discussed at:
https://discuss.python.org/t/adding-char-based-apis-for-unix/916/8
vstinner commentedon Mar 6, 2019
By the way, according to Nick Coghlan (author of the PEP-538), Py_Initialize() and Py_Main() called from an application embedding Python should not coerce the C locale.
ncoghlan commentedon Mar 6, 2019
They weren't *intended* to change it, and didn't in the original implementation of the PEP, but they do in the as-shipped Python 3.7 implementation, and I abandoned my attempts to revert to the as-designed behaviour as impractical given the other changes made for PEP-540. So that's a behavior we're stuck with now: they both have global side effects on the locale of the calling process.
ncoghlan commentedon Mar 14, 2019
Victor and I were discussing the appropriate behaviour for the "What do we do if _Py_PreInitialize() hasn't been called?" case, and Victor pointed out that the potential for mojibake provides a solid rationale for going back to the Python 3.6 behaviour: disable both C locale coercion *and* UTF-8 mode, and instead leave locale management to the embedding application.
That would also get us back to the originally intended behaviour of PEP-538, where it only happens by default in the CLI app, not the runtime support library. Back when I decided that doing that would be too complicated, the _Py_PreInitialize API didn't exist yet, and it didn't occur to me that the mojibake problem would affect UTF-8 mode as well.
vstinner commentedon Mar 27, 2019
New changeset d929f18 by Victor Stinner in branch 'master':
bpo-36443: Disable C locale coercion and UTF-8 Mode by default (GH-12589)
d929f18
vstinner commentedon Mar 27, 2019
My commit disabled C locale coercion and UTF-8 Mode by default when Python is embedded which fix this issue in Python 3.8. I close the issue.