Add locale.get_locale_encoding() and locale.get_current_locale_encoding()

[vstinner]

BPO	43552
Nosy	@malemburg, @vstinner, @methane, @eryksun
PRs	bpo-43552: Add locale.get_current_locale_encoding() #24931
Files	encodings.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-06-29.00:19:02.728>
created_at = <Date 2021-03-19.09:17:13.541>
labels = ['library', '3.10']
title = 'Add locale.get_locale_encoding() and locale.get_current_locale_encoding()'
updated_at = <Date 2021-06-29.00:19:02.727>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-06-29.00:19:02.727>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-29.00:19:02.728>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-03-19.09:17:13.541>
creator = 'vstinner'
dependencies = []
files = ['49894']
hgrepos = []
issue_num = 43552
keywords = ['patch']
message_count = 32.0
messages = ['389057', '389062', '389063', '389064', '389065', '389066', '389068', '389069', '389070', '389072', '389074', '389076', '389079', '389080', '389082', '389083', '389087', '389088', '389089', '389090', '389091', '389093', '389098', '389100', '389101', '389102', '389118', '389131', '389136', '389159', '389195', '396686']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'methane', 'eryksun']
pr_nums = ['24931']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43552'
versions = ['Python 3.10']

[vstinner]

I propose to add two new functions:

• locale.get_locale_encoding(): it's exactly the same than locale.getpreferredencoding(False).

• locale.get_current_locale_encoding(): always get the current locale encoding. Read the ANSI code page on Windows, or nl_langinfo(CODESET) on other platforms. Ignore the UTF-8 Mode. Don't always return "UTF-8" on macOS, Android, VxWorks.

Technically, locale.get_locale_encoding() would simply expose _locale.get_locale_encoding() that I added recently. It calls the new private _Py_GetLocaleEncoding() function (which has no argument).

By the way, Python requires nl_langinfo(CODESET) to be built. It's not a new requirement of Python 3.10, but I wanted to note that, I noticed it when I implemented _locale.get_locale_encoding() :-)

Python has a bad habit of lying to the user: locale.getpreferredencoding(False) is *NOT* the current locale encoding in multiple cases.

• locale.getpreferredencoding(False) always return "UTF-8" on macOS, Android and VxWorks
• locale.getpreferredencoding(False) always return "UTF-8" if the UTF-8 Mode is enabled
• otherwise, it returns the current locale encoding: ANSI code page on Windwos, or nl_langinfo(CODESET) on other platforms

Even if locale.getpreferredencoding(False) already exists, I propose to add locale.get_locale_encoding() because I dislike locale.getpreferredencoding() API. By default, this function sets temporarily LC_CTYPE to the user preferred locale. It can cause mojibake in other threads since setlocale(LC_CTYPE, "") affects all threads :-( Calling locale.getpreferredencoding(), rather than locale.getpreferredencoding(False), is not what most people expect. This API can be misused.

On the other side, locale.get_locale_encoding() does exactly what it says: only *get* the encoding, don't *set* temporarily a locale to something else.

By the way, the locale.localeconv() function can change temporarily LC_CTYPE locale to the LC_MONETARY locale which can cause other threads to use the wrong LC_CTYPE locale! But this is a different issue.

[malemburg]

On 19.03.2021 10:17, STINNER Victor wrote:

New submission from STINNER Victor <vstinner@python.org>:

I propose to add two new functions:

• locale.get_locale_encoding(): it's exactly the same than locale.getpreferredencoding(False).

• locale.get_current_locale_encoding(): always get the current locale encoding. Read the ANSI code page on Windows, or nl_langinfo(CODESET) on other platforms. Ignore the UTF-8 Mode. Don't always return "UTF-8" on macOS, Android, VxWorks.

I'm not sure whether this would improve the situation much.

The problem is that the locale module is meant to expose the lib C
locale settings, but many of the recent additions actually do something
completely different: they look into the process and user environment
and try to determine external settings, which are not reflected in
the lib C locale settings.

I had added locale.getdefaultlocale() to give applications a chance
to determine the locale setting defined by the process environment
*without* calling setlocale(LC_ALL, '') and causing problems
in other threads. I used the X11 database for locale encodings,
which was the closest you could get to in terms of a standard for
encodings at the time (around 2000).

Part of the return value is the encoding, which would be set.

Martin later added locale.getpreferredencoding(), which tries to
determine the encoding in a different way new way, based on
nl_langset(CODEINFO). As you mentioned, this intention was broken
on several platforms by forcing UTF-8 as output. And in many cases,
the API had to call setlocale() as well, causing the thread problems.

However, the problem with nl_langset(CODEINFO) is the same as
with setlocale(): it returns the current state of the lib C
settings, which may well point to the 'C' locale. Not the ones
the user has configured in the OS environment. So while you get
an encoding defined by lib C for the current locale settings
(without guessing it as with locale.getdefaultlocale()), you
still don't get what the user really wants to use.

Unfortunately, lib C does not provide a way to query the locale
database without changing the locale settings at the same time.
This is the main issue we're facing.

Now, the correct way in all this would be to just call
setlocale(LC_ALL, '') at the start of the application and
not try to apply all the magic to get around this. But this
has to be done by the application and not Python (which may
well be embedded into some other application).

I'd suggest to add a single new API:

locale.getencoding()

which interfaces to nl_langinfo(CODESET) or the Windows code
page and does not try to do any magic, ie. does *not* call
setlocale(). It needs to return what the lib C currently
knows and uses as encoding.

locale.getpreferredencoding() should then be deprecated.

It does not make sense to pretend to query information which is
not really directly available from the lib C locale system.

And the documentation should point out that applications should
call setlocale(LC_ALL, '') when they start up, if they want to
get the lib C locale, and thus Python locale module, setup to
work based on what the user really wants -- instead of just
guessing at this.

PS: The locale module normally does not use underscores in
function names, so it's not a good idea to add more.

[vstinner]

I created this issue while reviewing the implementation of the PEP-597: PR 19481.

Copy of my comments on the PR related to this issue.

_locale.get_locale_encoding() calls _Py_GetLocaleEncoding() which returns UTF-8 if the Python UTF-8 Mode is enabled.

Maybe the function could have a flag: please don't lie to me and return the current locale encoding ;-)

Or we could add a function to get the *current* locale encoding: **locale.get_current_locale_encoding()**.

This one would ignore the UTF-8 Mode and call nl_langinfo(CODESET). There are APIs to use the *current* locale encoding: PyUnicode_EncodeLocale/PyUnicode_DecodeLocale and _Py_EncodeLocaleEx/_Py_DecodeLocaleEx with current_locale=1. You can see which functions use it:

• decode tm_zone field of localtime_r() and gmtime()
• decode tzname[0] and tzname[1] strings
• decode setlocale() result
• decode some localeconv() fields (this function requires to switch to different locale encoding, it's bad!)
• decode nl_langinfo() result
• decode gettext(), dgettext(), dcgettext(), textdomain(), bindtextdomain(), bind_textdomain_codeset() result
• decode strerror() and dlerror() result
• encode/decode in the readline module
• encode format string for strftime() in time.strftime() (only used on Windows, Unix provides wcsftime) and then decode strftime() result

encoding="locale" : Uses locale encoding regardless UTF-8 mode.

Currently, open(encoding=None) doesn't work like that. For example, on macOS, Android and VxWorks, it always use UTF-8. And if the UTF-8 Mode is used, UTF-8 is used.

In the PEP-597, I read the encoding="locale" is the same than encoding=None but don't emit an EncodingWarning. Where the PEP-597 changes the chosen encoding for encoding=None case? The PEP says "locale encoding" without specifying exactly what it is. In Python, it means different things depending on the context. There is subtle difference the **current** locale encoding and "the locale encoding". I agree that it needs some clarification :-)

While we discuss encodings, I never understood why open() gets the current locale encoding from nl_langinfo(CODESET), encoding which can change at runtime while Python is running. For example, if thread A calls open(filename, encoding=None), thread B calls locale.localeconv(), and the LC_MONETARY locale uses a different encoding than the LC_CTYPE locale, thread A can get the LC_MONETARY encoding because of how locale.localeconv() is currently implemented: it changes temporarily LC_CTYPE to LC_MONETARY to decode the monetary fields of localeconv() result.

I would prefer that Python uses the same encoding for the whole lifetime of the process, since the beginning until the end. The Python filesystem encoding is a good choice for that. It's the same than locale.getpreferredencoding(False) (currently used by open() and friends), but becomes different if the LC_CTYPE is changed (temporarily or permanently).

[vstinner]

I created PR 24931 to add locale.get_current_locale_encoding(). I tried to clarified the differences between the "current locale encoding" and the "locale encoding".

Maybe we should rename the "locale encoding" to the "Python locale encoding", since it's not what most Unix developers would expect. What do you think?

While most locale function have no underscore in their name, it seems like the current trend is to allow underscores in names for *new* functions. For example, the sys module has without underscores:

• sys.getallocatedblocks()
• sys.getdefaultencoding()
• sys.getfilesystemencodeerrors
• ...

But it got new functions with underscores:

• sys.set_asyncgen_hooks()
• sys.set_coroutine_origin_tracking_depth()

... and there are some old functions with underscores:

• sys.exc_info()
• sys.call_tracing()
• sys._clear_type_cache()
• sys._current_frames()

In the locale module, there is one existing function with an undercore:

• locale.format_string()

[vstinner]

Now, the correct way in all this would be to just call setlocale(LC_ALL, '') at the start of the application

Python now does that during its initialization on all platforms. So getpreferredencoding(False) is what its documentation says: the user preferred encoding, the LC_CTYPE locale encoding.

On Python 3.7, _Py_SetLocaleFromEnv(LC_CTYPE) was called in _Py_InitializeCore() on Unix, but not on Windows.

Since Python 3.8, _PyPreConfig_Write() calls _Py_SetLocaleFromEnv(LC_CTYPE) on all platforms including Windows. See bpo-34485 and my article for more details ("C locale on Windows" section):
https://vstinner.github.io/python3-locales-encodings.html

_Py_SetLocaleFromEnv(LC_CTYPE) calls setlocale(LC_CTYPE, ""), but has more complex code on Android.

[vstinner]

locale.getencoding()

which interfaces to nl_langinfo(CODESET) or the Windows code
page and does not try to do any magic, ie. does *not* call
setlocale(). It needs to return what the lib C currently
knows and uses as encoding.

This is locale.get_current_locale_encoding(). I would like to put "current" in the name, because there is a lot of confusion between get_current_locale_encoding() encoding and locale.getpreferredencoding(False) encoding. In locale.getpreferredencoding(False), Python ignores the locale in some cases which is counter intuitive.

I propose to add new functions to reduce confusion and better document the subtle differences between the different "locale encodings".

That's also why I propose to rename the "locale encoding" to the "Python locale encoding" in the documentation: clarify the Python ignores the locale sometimes.

The PEP-538 (coerce the C locale) and PEP-540 (Python UTF-8 Mode) introduced confusion.

[malemburg]

On 19.03.2021 11:36, STINNER Victor wrote:

STINNER Victor <vstinner@python.org> added the comment:

> locale.getencoding()
>
> which interfaces to nl_langinfo(CODESET) or the Windows code
> page and does not try to do any magic, ie. does *not* call
> setlocale(). It needs to return what the lib C currently
> knows and uses as encoding.

This is locale.get_current_locale_encoding(). I would like to put "current" in the name, because there is a lot of confusion between get_current_locale_encoding() encoding and locale.getpreferredencoding(False) encoding. In locale.getpreferredencoding(False), Python ignores the locale in some cases which is counter intuitive.

These attempts have resulted much of the confusion around the locale
module. It's better not to create more of it.

• "locale" in the name is unnecessary, since this is the locale module.

• If you add "current", people will rightly ask: then what do all the
  other APIs in the locale module return ? Of course, they all return
  the current state of settings :-) So this is unnecessary as well.

locale.getencoding() works in the same way as locale.getlocale().
It interfaces to the lib C and returns the current encoding setting
as known by the lib C. It's just a more intuitive name than
locale.nl_langinfo(CODESET) and works on Windows as well.

And, again, locale.getpreferredencoding() should be deprecated.
The API has been misused in too many ways and is completely broken
by now. It was a good idea at the time, when Martin added it,
even though I never liked the name.

[vstinner]

Attached encodings.py lists the different "locale encodings" used by Python. Example:
---

$ LANG=fr_FR ./python -X utf8 encodings.py fr_FR@euro
Set LC_CTYPE to 'fr_FR@euro'

LC_ALL env var: ''
LC_CTYPE env var: ''
LANG env var: 'fr_FR'
LC_CTYPE locale: 'fr_FR@euro'
Coerce C locale: 0
Python UTF-8 Mode: 1

(1) Python FS encoding
sys.getfilesystemencoding(): 'utf-8'

(2) Python locale encoding
_locale._get_locale_encoding(): 'UTF-8'
locale.getpreferredencoding(False): 'UTF-8'

(3) Current locale encoding
locale.get_current_locale_encoding(): 'ISO-8859-15'

(4) And more encodings for more fun!
locale.getdefaultlocale()[1]: 'ISO8859-1'
locale.getpreferredencoding(True): 'UTF-8'
---

Python starts with LC_CTYPE locale set to fr_FR (ISO8859-1), then the script sets the LC_CTYPE locale to fr_FR@euro (ISO-8859-15). The Python UTF-8 Mode is enabled explicitly. We get a funny combination of not less than 3 encodings!

• UTF-8
• ISO-8859-1
• ISO-8859-15

Which one is the correct one? Wel... It depends :-)

(1) The Python filesystem encoding is used to call almost all operating system functions: encode to the OS and decode from the OS. Filenames, environment variables, command line options, etc.

(2) The "Python" locale encoding is used by open() when no encoding is specific.

(3) The current locale encoding is used for a limited amount of functions that I listed in msg389063. Most users should not use it.

(4) locale.getpreferredencoding(True) is a weird beast. It is Python locale encoding until setlocale(LC_CTYPE, locale) is called for the first time. But it can be same if the Python UTF-8 Mode is enabled. I'm not sure in which category we should put this function :-(

(4 bis) locale.getdefaultlocale()[1] is the only function returning the ISO-8859-1 encoding. This encoding is not used by any function. I'm not sure of the purpose of this function. It sounds confusing.

I suggest to deprecate locale.getpreferredencoding(True).

I'm not sure what to do with locale.getdefaultlocale(). Should we deprecate it? I never used this function. How is it used? For which purpose?

I undertand that in 2000, locale.getdefaultlocale() was interesting to avoid calling setlocale(LC_CTYPE, ""). But Python 3 calls setlocale(LC_CTYPE, "") by default at startup since the early versions, and it's now called on all platforms since Python 3.8. Moreover, its internal database seems to be outdated and is painful to maintain (especially if we consider all platforms supported by Python, not only Linux, there are many issues on macOS).

[vstinner]

Martin later added locale.getpreferredencoding(), which tries to
determine the encoding in a different way new way, based on
nl_langset(CODEINFO). As you mentioned, this intention was broken
on several platforms by forcing UTF-8 as output.

When I designed and implemented the PEP-540 (Python UTF-8 Mode), I tried to leave getpreferredencoding() unchanged. The problem was that I quickly got mojibake because too many functions call getpreferredencoding(False):

• open() and _pyio.open() -- in Python 3.10, open() now calls the C _Py_GetLocaleEncoding() function to fix issues during Python shutdown, it also avoids issues at startup.
• Many gettext functions
• cgi to decode the query string from QUERY_STRING env var or sys.argv[1]}
• xml.etree.ElementTree.write(encoding="unicode") is some cases

The Python UTF-8 Mode ignores the locale *on purpose*. But I agree that it's surprising and can lead to confusion. That's what I'm trying to fix here :-)