Repository navigation
encodings: the "mbcs" alias doesn't work #90826
Description
| BPO | 46668 |
|---|---|
| Nosy | @vstinner, @eryksun |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2022-02-23.17:14:28.416>
created_at = <Date 2022-02-06.23:06:49.042>
labels = ['library', '3.11']
title = 'encodings: the "mbcs" alias doesn\'t work'
updated_at = <Date 2022-02-23.17:14:28.415>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2022-02-23.17:14:28.415>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-02-23.17:14:28.416>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2022-02-06.23:06:49.042>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46668
keywords = ['patch']
message_count = 8.0
messages = ['412678', '412680', '412683', '412691', '412738', '412777', '412847', '413825']
nosy_count = 2.0
nosy_names = ['vstinner', 'eryksun']
pr_nums = ['31174']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue46668'
versions = ['Python 3.11']
Activity
vstinner commentedon Feb 6, 2022
While working on bpo-46659, I found a bug in the encodings "mbcs" alias. Even if the function has 2 tests (in test_codecs and test_site), both tests missed the bug :-(
I fixed the alias with this change:
---
commit 04dd60e
Author: Victor Stinner <vstinner@python.org>
Date: Sun Feb 6 21:50:09 2022 +0100
---
But Eryk found two bugs:
"""
This was never true before. With 1252 as my ANSI code page, I checked codecs.lookup('cp1252') in 2.7, 3.4, 3.5, 3.6, 3.9, and 3.10, and none of them return the "mbcs" encoding. It's not equivalent, and not supposed to be. The implementation of "cp1252" should be cross-platform, regardless of whether we're on a Windows system with 1252 as the ANSI code page, as opposed to a Windows system with some other ANSI code page, or a Linux or macOS system.
The differences are that "mbcs" maps every byte, whereas our code-page encodings do not map undefined bytes, and the "replace" handler of "mbcs" uses a best-fit mapping (e.g. "α" -> "a") when encoding text, instead of mapping all undefined characters to "?".
"""
and my new test fails if PYTHONUTF8=1 env var is set:
"""
This will fail if PYTHONUTF8 is set in the environment, because it overrides getpreferredencoding(False) and _get_locale_encoding().
"""
The code for the "mbcs" alias changed at lot between Python 3.5 and 3.7.
In Python 3.5, site module:
---
In Python 3.6, encodings module:
---
(...)
codecs.register(search_function)
Python 3.7, encodings module:
---
(...)
codecs.register(search_function)
The Python 3.6 and 3.7 "codecs.register(_alias_mbcs)" doesn't work because "search_function()" is tested before and it works for "cpXXX" encodings. My changes changes the order in which codecs search functions are registered: first the MBCS alias, then the encodings search_function().
In Python 3.5, the alias was only created if Python didn't support the code page.
vstinner commentedon Feb 6, 2022
The alias was created in 2003 to fix bpo-671666:
---
commit 4eab486
Author: Martin v. Löwis <martin@v.loewis.de>
Date: Mon Mar 3 09:34:01 2003 +0000
---
In 2003, bpo-671666 was created because Python didn't support "cp932" encoding, whereas the MBCS codec was available and could used directly since cp932 was the ANSI code page.
The alias allows to support the ANSI code 932 without implement it.
But Python got a "cp932" codec the year after:
---
commit 3e2a306
Author: Hye-Shik Chang <hyeshik@gmail.com>
Date: Sat Jan 17 14:29:29 2004 +0000
---
vstinner commentedon Feb 6, 2022
Python 3.11 supports the 40 code pages:
eryksun commentedon Feb 7, 2022
Isn't the 3.6-3.10 ordering of search_function() and _alias_mbcs() correct as a fallback? In this case, Python doesn't support a cross-platform encoding for the code page. That's why the old implementation of test_mbcs_alias() mocked _winapi.GetACP() to return 123 and then checked that looking up 'cp123' returned the "mbcs" codec.
I'd actually prefer to extend this by implementing _winapi.GetOEMCP() and using "oem" as a fallback for that case.
vstinner commentedon Feb 7, 2022
I don't think that this fallback is needed anymore. Which Windows code page can be used as ANSI code page which is not already implemented as a Python codec?
eryksun commentedon Feb 7, 2022
Python has full coverage of the ANSI and OEM code pages in the standard Windows locales, but I don't have any experience with custom (i.e. supplemental or replacement) locales.
https://docs.microsoft.com/en-us/windows/win32/intl/custom-locales
Here's a simple script to check the standard locales.
Output:
Some locales are Unicode only (e.g. Hindi-India) or have no OEM code page, which the above code skips by checking for "0" or "1" as the code page value. Windows 10+ allows setting the system locale to a Unicode-only locale, for which it uses UTF-8 (65001) for ANSI and OEM.
The OEM code page matters because the console input and output code pages default to OEM, e.g. for os.device_encoding(). The console's I/O code pages are used in Python by low-level os.read() and os.write(). Note that the console doesn't properly implement using UTF-8 (65001) as the input code page. In this case, input read from the console via ReadFile() or ReadConsoleA() has a null byte in place of each non-ASCII character.
vstinner commentedon Feb 8, 2022
I created #75401 which basically restores Python 3.10 code but enhances the test.
vstinner commentedon Feb 23, 2022
commit ccbe804
Author: Victor Stinner <vstinner@python.org>
Date: Tue Feb 22 22:04:07 2022 +0100