[Windows] test_locale.TestMiscellaneous.test_getsetlocale_issue1813() fails

[tjguk]

BPO	37945
Nosy	@terryjreedy, @pfmoore, @db3l, @vstinner, @tjguk, @jkloth, @ned-deily, @methane, @ambv, @zware, @serhiy-storchaka, @eryksun, @zooba, @miss-islington, @tirkarthi
PRs	bpo-37945: Fix test_locale.test_getsetlocale_issue1813() #25110 [3.9] bpo-37945: Fix test_locale.test_getsetlocale_issue1813() (GH-25110) #25112 [3.8] bpo-37945: Fix test_locale.test_getsetlocale_issue1813() (GH-25110) #25113
Files	pythoninfo.txt: python -mtest.pythoninfo

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tjguk'
closed_at = <Date 2021-03-31.12:08:34.105>
created_at = <Date 2019-08-25.17:33:21.926>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = '[Windows] test_locale.TestMiscellaneous.test_getsetlocale_issue1813() fails'
updated_at = <Date 2021-03-31.22:47:06.662>
user = 'https://github.com/tjguk'

bugs.python.org fields:

activity = <Date 2021-03-31.22:47:06.662>
actor = 'vstinner'
assignee = 'tim.golden'
closed = True
closed_date = <Date 2021-03-31.12:08:34.105>
closer = 'vstinner'
components = ['Library (Lib)', 'Windows']
creation = <Date 2019-08-25.17:33:21.926>
creator = 'tim.golden'
dependencies = []
files = ['48561']
hgrepos = []
issue_num = 37945
keywords = ['patch']
message_count = 38.0
messages = ['350466', '350470', '350471', '350485', '350491', '350510', '350548', '350549', '350559', '350568', '350569', '350571', '350573', '350574', '350598', '350820', '350823', '351788', '361804', '361806', '379068', '382739', '389005', '389657', '389661', '389784', '389839', '389867', '389883', '389884', '389885', '389886', '389888', '389890', '389891', '389892', '389938', '389940']
nosy_count = 16.0
nosy_names = ['terry.reedy', 'paul.moore', 'db3l', 'vstinner', 'tim.golden', 'jkloth', 'ned.deily', 'methane', 'guy.linton', 'lukasz.langa', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'miss-islington', 'xtreak']
pr_nums = ['25110', '25112', '25113']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37945'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[tjguk]

On a Win10 machine I'm consistently seeing test_locale (and test__locale) fail. I'll attach pythoninfo.

======================================================================
ERROR: test_getsetlocale_issue1813 (test.test_locale.TestMiscellaneous)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Users\tim\work-in-progress\cpython\lib\test\test_locale.py", line 531, in test_getsetlocale_issue1813
    locale.setlocale(locale.LC_CTYPE, loc)
  File "C:\Users\tim\work-in-progress\cpython\lib\locale.py", line 604, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

[tjguk]

Ok; so basically this doesn't work:

<code>
import locale
locale.setlocale(locale.LC_CTYPE, locale.getdefaultlocale())
</code>

It gives "locale.Error: unsupported locale setting" which comes from https://github.com/python/cpython/blob/master/Modules/_localemodule.c#L107

(For locale.getdefaultlocale() you could substitute locale.getlocale() or simply ("en_GB", "cp1252")). On my machine it raises that exception on Python 2.7.15, 3.6.6 and on master.

Interestingly, none of the other tests in test_locale appear to exercise the 2-tuple 2nd param to setlocale. When you call setlocale and it returns the previous setting, it's a single string, eg "en_GB" etc. Passing that back in works. But when you call getlocale, it returns the 2-tuple, eg ("en_GB", "cp1252"). But all the other tests use the setlocale-returns-current trick for their setup/teardown.

I've quickly tested on 3.5 on Linux and the 2-tuple version works ok. I assume it's working on buildbots or we'd see the Turkish test failing every time. So is there something different about my C runtime, I wonder?

[tjguk]

Just to save you looking, the code in https://github.com/python/cpython/blob/master/Modules/_localemodule.c#L107 converts the 2-tuple to lang.encoding form so the C module is seeing "en_GB.cp1252"

[eryksun]

local.normalize is generally wrong in Windows. It's meant for POSIX systems. Currently "tr_TR" is parsed as follows:

    >>> locale._parse_localename('tr_TR')
    ('tr_TR', 'ISO8859-9')

The encoding "ISO8859-9" is meaningless to Windows. Also, the old CRT only ever supported either full language/country names or non-standard abbreviations -- e.g. either "Turkish_Turkey" or "trk_TUR". Having locale.getdefaultlocale() return ISO two-letter codes (e.g. "en_GB") was fundamentally wrong for the old CRT. (2.7 will die with this wart.)

3.5+ uses the Universal CRT, which does support standard ISO codes, but only in BCP 47 [1] locale names of the following form:

language           ISO 639
["-" script]       ISO 15924
["-" region]       ISO 3166-1

BCP 47 locale names have been preferred by Windows for the past 13 years, since Vista was released. Windows extends BCP 47 with a non-standard sort-order field (e.g. "de-Latn-DE_phoneb" is the German language with Latin script in the region of Germany with phone-book sort order). Another departure from strict BCP 47 in Windows is allowing underscore to be used as the delimiter instead of hyphen.

In a concession to existing C code, the Universal CRT also supports an encoding suffix in BCP 47 locales, but this can only be either ".utf-8" or ".utf8". (Windows itself does not support specifying an encoding in a locale name, but it's Unicode anyway.) No other encoding is allowed. If ".utf-8" isn't specified, a BCP 47 locale defaults to the locale's ANSI codepage. However, there's no way to convey this in the locale name itself. Also, if a locale is Unicode only (e.g. Hindi), the CRT implicitly uses UTF-8 even without the ".utf-8" suffix.

The following are valid BCP 47 locale names in the CRT: "tr", "tr.utf-8", "tr-TR", "tr_TR", "tr_TR.utf8", or "tr-Latn-TR.utf-8". But note that "tr_TR.1254" is not supported.

The following shows that omitting the optional "utf-8" encoding in a BCP 47 locale makes the CRT default to the associated ANSI codepage.

    >>> locale.setlocale(locale.LC_CTYPE, 'tr_TR')
    'tr_TR'
    >>> ucrt.___lc_codepage_func()
    1254

C ___lc_codepage_func() queries the codepage of the current locale. We can directly query this codepage for a BCP 47 locale via GetLocaleInfoEx:

    >>> cpstr = (ctypes.c_wchar * 6)()
    >>> kernel32.GetLocaleInfoEx('tr-TR',
    ...     LOCALE_IDEFAULTANSICODEPAGE, cpstr, len(cpstr))
    5
    >>> cpstr.value
    '1254'

If the result is '0', it's a Unicode-only locale (e.g. 'hi-IN' -- Hindi, India). Recent versions of the CRT use UTF-8 (codepage 65001) for Unicode-only locales:

    >>> locale.setlocale(locale.LC_CTYPE, 'hi-IN')
    'hi-IN'
    >>> ucrt.___lc_codepage_func()
    65001

Here are some example locale tuples that should be supported, given that the CRT continues to support full English locale names and non-standard abbreviations, in addition to the new BCP 47 names:

('tr', None)
('tr_TR', None)
('tr_Latn_TR, None)
('tr_TR', 'utf-8')

('trk_TUR', '1254')
('Turkish_Turkey', '1254')

The return value from C setlocale can be normalized to replace hyphen delimiters with underscores, and "utf8" can be normalized as "utf-8". If it's a BCP 47 locale that has no encoding, GetLocaleInfoEx can be called to query the ANSI codepage. UTF-8 can be assumed if it's a Unicode-only locale.

As to prefixing a codepage with 'cp', we don't really need to do this. We have aliases defined for most, such as '1252' -> 'cp1252'. But if the 'cp' prefix does get added, then the locale module should at least know to remove it when building a locale name from a tuple.

[1] https://tools.ietf.org/rfc/bcp/bcp47.txt

[tjguk]

Thanks, Eryk. Your explanation is as clear as always. But my question is, then: why is my machine failing this test [the only one which uses this two-part locale] and not the buildbots or (presumably) any other Windows developer?

[eryksun]

But my question is, then: why is my machine failing this test [the
only one which uses this two-part locale] and not the buildbots or
(presumably) any other Windows developer?

test_getsetlocale_issue1813 fails for me as well. I can't imagine how setlocale(LC_CTYPE, "tr_TR.ISO8859-9") would succeed with recent versions of the Universal CRT in Windows. It parses "tr_TR" as a BCP 47 locale name, which only supports UTF-8 (e.g. "tr_TR.utf-8") and implicit ANSI (e.g. "tr_TR"). Plus "ISO8859-9" in general isn't a supported encoding of the form ".<codepage>", ".ACP" (ANSI), ".utf8", or ".utf-8".

With the old CRT (2.x and <=3.4) and older versions of the Universal CRT, the initial locale.setlocale(locale.LC_CTYPE 'tr_TR') call fails as an unsupported locale, so the test is skipped:

test_getsetlocale_issue1813 (__main__.TestMiscellaneous) ... skipped 'test needs Turkish locale'

The old CRT only supports "trk_TUR", "trk_Turkey", "turkish_TUR", and "turkish_Turkey".

[zooba]

So is the fix here to update locale._build_localename to check something like this?

if encoding is None:
    return language
elif sys.platform == 'win32' and encoding not in {'utf8', 'utf-8'}:
    return language
else:
    return language + '.' + encoding

[tjguk]

I agree that that could be a fix. And certainly, if it turns out that this could never have (recently) worked as Eryk is suggesting, then let's go for it.

But I still have this uneasy feeling that it's not failing on the buildbots and I can't see any sign of a skipped test in the test stdio. I just wonder whether there's something else at play here.

[zooba]

I pushed a custom buildbot run that only runs this test in verbose mode, and it looks like the test is being skipped some other way?

https://buildbot.python.org/all/#/builders/48/builds/36
https://buildbot.python.org/all/#/builders/42/builds/54

I don't see any evidence there that it's running at all, though I do on my own machine.

Perhaps one of the other buildbot settings causes it to run in a different order and something skips the entire class? I haven't dug in enough to figure that out yet.

[eryksun]

We get into trouble with test_getsetlocale_issue1813 because normalize() maps "tr_TR" (supported) to "tr_TR.ISO8859-9" (not supported).

    >>> locale.normalize('tr_TR')
    'tr_TR.ISO8859-9'

We should skip normalize() in Windows. It's based on a POSIX locale_alias mapping that can only cause problems. The work for normalizing locale names in Windows is best handled inline in _build_localename and _parse_localename.

For the old long form, C setlocale always returns the codepage encoding (e.g. "Turkish_Turkey.1254") or "utf8", so that's simple to parse. For BCP 47 locales, the encoding is either "utf8" or "utf-8", or nothing at all. For the latter, there's an implied legacy ANSI encoding. This is used by the CRT wherever we depend on byte strings, such as in time.strftime:

mojibake:

    >>> locale.setlocale(locale.LC_CTYPE, 'en_GB')
    'en_GB'
    >>> time.strftime("\u0100")
    'A'

correct:

    >>> locale.setlocale(locale.LC_CTYPE, 'en_GB.utf-8')
    'en_GB.utf-8'
    >>> time.strftime("\u0100")
    'Ā'

(We should switch back to using wcsftime if possible.)

The implicit BCP-47 case can be parsed as None -- e.g. ("tr_TR", None). However, it might be useful to support getting the ANSI codepage via GetLocaleInfoEx [1]. A high-level function in locale could internally call _locale.getlocaleinfo(locale_name, LOCALE_IDEFAULTANSICODEPAGE). This would return a string such as "1254". or "0" for a Unicode-only language.

For _build_localename, we can't simply limit the encoding to UTF-8. We need to support the old long/abbreviated forms (e.g. "trk_TUR", "turkish_Turkey") in addition to the newer BCP 47 locale names. In the old form we have to support the following encodings:

* codepage encodings, with an optional "cp" prefix that has 
  to be stripped, e.g. ("trk_TUR", "cp1254") -> "trk_TUR.1254"
* "ACP" in upper case only -- for the ANSI codepage of the 
  language
* "utf8" (mixed case) and "utf-8" (mixed case)

(The CRT documentation says "OEM" should also be supported, but it's not.)

A locale name can also omit the language in the old form -- e.g. (None, "ACP") or (None, "cp1254"). The CRT uses the current language in this case. This is discouraged because the result may be nonsense.

[1] https://docs.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getlocaleinfoex