strptime fails in non-UTF locale

[pitrou]

BPO	5905
Nosy	@malemburg, @terryjreedy, @pitrou, @vstinner, @ezio-melotti
Files	tzname_encoding.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-12-09.19:19:41.033>
created_at = <Date 2009-05-02.15:17:20.511>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'strptime fails in non-UTF locale'
updated_at = <Date 2011-12-09.19:19:41.033>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2011-12-09.19:19:41.033>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-12-09.19:19:41.033>
closer = 'vstinner'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2009-05-02.15:17:20.511>
creator = 'pitrou'
dependencies = []
files = ['23881']
hgrepos = []
issue_num = 5905
keywords = ['patch']
message_count = 10.0
messages = ['86953', '97808', '98105', '112997', '113736', '138210', '138722', '149024', '149038', '149115']
nosy_count = 7.0
nosy_names = ['lemburg', 'terry.reedy', 'pitrou', 'vstinner', 'ezio.melotti', 'BreamoreBoy', 'python-dev']
pr_nums = []
priority = 'critical'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5905'
versions = ['Python 3.2', 'Python 3.3']

[pitrou]

time.strptime() fails with non-UTF8 locales, *even when the input is
totally ASCII*.

>>> locale.setlocale(locale.LC_TIME, "fr_FR.ISO8859-15")
'fr_FR.ISO8859-15'
>>> time.strptime("2009-01-01", "%Y-%m-%d")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/py3k/__svn__/Lib/_strptime.py", line 461, in
_strptime_time
    return _strptime(data_string, format)[0]
  File "/home/antoine/py3k/__svn__/Lib/_strptime.py", line 307, in _strptime
    _TimeRE_cache = TimeRE()
  File "/home/antoine/py3k/__svn__/Lib/_strptime.py", line 188, in __init__
    self.locale_time = LocaleTime()
  File "/home/antoine/py3k/__svn__/Lib/_strptime.py", line 72, in __init__
    self.__calc_month()
  File "/home/antoine/py3k/__svn__/Lib/_strptime.py", line 98, in
__calc_month
    a_month = [calendar.month_abbr[i].lower() for i in range(13)]
  File "/home/antoine/py3k/__svn__/Lib/_strptime.py", line 98, in <listcomp>
    a_month = [calendar.month_abbr[i].lower() for i in range(13)]
  File "/home/antoine/py3k/__svn__/Lib/calendar.py", line 60, in __getitem__
    return funcs(self.format)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-3:
invalid data

[malemburg]

The reason for this is that the strftime() C lib API is used to build localized month names. With your setting, you'll get French Latin-1 month names and those cannot be coerced to UTF-8 due to the accented characters in them.

This works in Python 2.x since PyUnicode_FromString() et al. convert Latin-1 strings to Unicode.

Apparently, this was changed in Python 3.x without looking at the header file or looking at the Python 2.x implementation which mandate Latin-1 as input encoding. Even the Python 3.x header still says that PyUnicode_FromString() will convert from Latin-1 to Unicode.

No idea why time.strptime() even bothers with these month names, though, since neither the format string nor the string being parsed contains literal month names.

[vstinner]

I'm unable to reproduce the error. I tried locales fr_FR.iso88591 and fr_FR.iso885915@euro (fr_FR@euro), but the example works correctly. Should the terminal use the specified locale? My terminal uses fr_FR.utF8 locale. Should set_locale() be called before loaded time and/or calendar module?

[BreamoreBoy]

I can't reproduce this on Windows Vista with 3.1 or 3.2 despite trying several Western & Eastern European, Chinese & Japanese locales.

[vstinner]

I can't reproduce this on Windows ...

This issue is (was?) maybe specific to Linux.

[terryjreedy]

Still a problem in 3.2.1 or 3.3?

[vstinner]

I close the issue because I am unable to reproduce it.

[vstinner]

Oh! I think that I understood the problem: if HAVE_WCSFTIME is not defined, timemodule.c uses strftime(), instead of wcsftime(), encode input format and decode the format. It uses UTF-8 to encode/decode, whereas the right encoding is the locale encoding. Attached patch should fix this issue.

@antoine: Do you have any idea why HAVE_WCSFTIME was not defined?

wcsftime() is defined in <wchar.h> on Ubuntu. In configure, it is tested using AC_CHECK_FUNCS(wcsftime)