locale.getdefaultlocale() fails on Mac OS X with default language set to English

[DmitryJemerov]

BPO	18378
Nosy	@malemburg, @loewis, @barry-scott, @ronaldoussoren, @ncoghlan, @mattheww, @ned-deily, @bitdancer, @larryv, @serhiy-storchaka, @wm75, @Kentzo, @tsparber, @karolyi, @miss-islington, @BoboTiG, @rfrail3
PRs	bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename #14736 [3.8] bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736) #15569 [3.7] bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736) #15570
Files	getdefaultlocale.patch: Patch with tests issue-18378-py27.txt issue-18378-py35.txt issue18378-2015-07-25-py36.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-29.06:30:06.592>
created_at = <Date 2013-07-06.12:19:03.656>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'locale.getdefaultlocale() fails on Mac OS X with default language set to English'
updated_at = <Date 2019-08-29.06:30:06.586>
user = 'https://bugs.python.org/DmitryJemerov'

bugs.python.org fields:

activity = <Date 2019-08-29.06:30:06.586>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-29.06:30:06.592>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2013-07-06.12:19:03.656>
creator = 'Dmitry.Jemerov'
dependencies = []
files = ['30807', '39384', '39385', '40014']
hgrepos = []
issue_num = 18378
keywords = ['patch', 'needs review']
message_count = 45.0
messages = ['192422', '192429', '192433', '192445', '192446', '192447', '192460', '192820', '192821', '192822', '192827', '209731', '214397', '214555', '214556', '214564', '215215', '239485', '239702', '243262', '247318', '247322', '247326', '247333', '247335', '247338', '247339', '247418', '263659', '268579', '278540', '285317', '285318', '285319', '285329', '285360', '285370', '296829', '333371', '347803', '347805', '350706', '350709', '350710', '350731']
nosy_count = 19.0
nosy_names = ['lemburg', 'loewis', 'barry-scott', 'ronaldoussoren', 'ncoghlan', 'mattheww', 'ned.deily', 'r.david.murray', 'Dmitry.Jemerov', 'larryv', 'serhiy.storchaka', 'wolma', 'Ilya.Kulakov', 'tsparber', 'karolyi', 'alexander.sturm', 'miss-islington', 'Tiger-222', 'rfrail3']
pr_nums = ['14736', '15569', '15570']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18378'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[DmitryJemerov]

On Mac OS X 10.8 with the default language set to English (System Preferences | Language and Text), the default terminal application sets the LC_CTYPE environment variable to "UTF-8". If you run Python from the terminal and try to use locale.getdefaultlocate(), you get the following error:

> python
Python 2.7.2 (default, Oct 11 2012, 20:14:37)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 496, in getdefaultlocale
    return _parse_localename(localename)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 428, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8

(The stacktrace is from Python 2.7 but Python 3.3 suffers from the same problem.)

There are numerous workarounds for this problem (turning off the "Set locale environment variables on startup" option in the terminal settings, or adding "export LC_CTYPE=en_US.UTF8" to .bash_profile, selecting a language other than English in the Language & Text settings), but these require additional configuration from the user's side.

I think that the more useful behavior is for Python to handle this behavior of the system and not crash, even though it doesn't strictly comply to the POSIX standard.

The attached patch (against current Python 3.4 master branch) is one possible fix.

[ronaldoussoren]

Strange, I have LANG=en_US.UTF-8 in my environment and no LC_CTYPE. A clean test account does have the same behavior as you are seeing.

[ronaldoussoren]

The UTF-8 value seems suspect to me, but is actually supported by the system, changing it to a nonsense value results in failure in the C function setlocale.

As for the patch: I'd add this workaround only to the OSX platform (that is, test for sys.platform == 'darwin' before checking for UTF-8 as a value).

[DmitryJemerov]

Judging from the results of Googling for the error message, I'm far from the only one seeing this problem.

What exactly would be the benefit of adding the code to check for the platform?

[ronaldoussoren]

The test for darwin is needed because other platforms don't support "UTF-8" as a valid LC_CTYPE name, on a recent linux box:

>>> locale.setlocale(locale.LC_CTYPE, "UTF-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python2.7/lib/python2.7/locale.py", line 539, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

(And just calling setlocale to check if the value is valid is not an option because that changes process-global state)

[DmitryJemerov]

Why exactly does this matter? UTF-8 not being a valid LC_CTYPE value simply means that no one running Linux will ever have LC_CTYPE set to UTF-8, and the branch will never be hit.

OTOH, adding the check will make the code harder to test and simply larger (no code is always better than any non-zero amount of code).

[DmitryJemerov]

A related issue (with a patch that touches the same locale parsing code) is http://bugs.python.org/issue5815

[loewis]

Why do you need the "getdefaultlocale" function in the first place? I'd advise against using it, precisely because it can trigger problems like this one.

[DmitryJemerov]

I personally don't, but the function is used by Sphinx, which is what I was trying to get to work when I ran into this problem.

[bitdancer]

Regardless of the resolution here, the use of getdefaultlocale could be reported as a bug on the sphinx tacker.

[loewis]

FWIW, I couldn't find any use of getdefaultlocale in any of the hg revisions (using hg grep) in

https://bitbucket.org/birkenfeld/sphinx/

Instead, it's (probably) docutils, which has this code:

    locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
    # locale.getpreferredencoding([do_setlocale=True|False])
    # has side-effects | might return a wrong guess.
    # (cf. Update 1 in http://stackoverflow.com/questions/4082645/using-python-2-xs-locale-module-to-format-numbers-and-currency)

I find that quite unfortunate, since locale.getpreferredencoding() would have don the right thing (IMO).

[ronaldoussoren]

I just ran into this problem myself.

On fresh installs of OSX 10.9 LC_CTYPE is set to "UTF-8" (at least for english language users), and now sphinx won't work :-(

Is Dimitrys patch acceptable (either as is, or with my suggestion of checking for sys.platform == "darwin")?

[ned-deily]

Ronald or Dmitry, can you elaborate under what conditions you start your login shell on 10.9? I cannot reproduce the behavior you observe. With 10.9 Terminal.app and the default language settings in System Preferences and with the default Terminal.app preferences, specifically Settings -> (Profile) -> Advanced -> Character encoding -> Unicode (UTF-8) and "Set LANG environment variable on startup" checked, login sessions have LANG=en_US.UTF-8 defined and LC_CTYPE is not defined at all. Are you sure that isn't begin created by a shell profile somewhere? (I can't check earlier OS X releases at the moment.) That said, I agree that, if OS X accepts "UTF-8" as a valid locale, the locale module should, too.