_localemodule.c: str2uni() with different LC_NUMERIC and LC_CTYPE

[skrah]

BPO	7442
Nosy	@loewis, @mdickinson, @vstinner, @ericvsmith, @mcepl, @skrah
Files	set_ctype_before_mbstowcs.patch inconsistent_locale_encodings.py mbstowcs_l.c

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-10-14.19:30:57.384>
created_at = <Date 2009-12-05.10:44:18.463>
labels = ['type-bug']
title = '_localemodule.c: str2uni() with different LC_NUMERIC and LC_CTYPE'
updated_at = <Date 2018-01-10.15:46:42.607>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2018-01-10.15:46:42.607>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-10-14.19:30:57.384>
closer = 'vstinner'
components = []
creation = <Date 2009-12-05.10:44:18.463>
creator = 'skrah'
dependencies = []
files = ['16221', '32303', '32306']
hgrepos = []
issue_num = 7442
keywords = ['patch']
message_count = 35.0
messages = ['95988', '96008', '96534', '96535', '96544', '96556', '96557', '99317', '190055', '200892', '200912', '200916', '200917', '200927', '200928', '200929', '200933', '200944', '200954', '200957', '200960', '200961', '200968', '200969', '200970', '200974', '200976', '200983', '202117', '202127', '202132', '202145', '229318', '229338', '309770']
nosy_count = 6.0
nosy_names = ['loewis', 'mark.dickinson', 'vstinner', 'eric.smith', 'mcepl', 'skrah']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7442'
versions = ['Python 3.2']

[skrah]

Hi, the following works in 2.7 but not in 3.x:

>>> import locale
>>> from decimal import *
>>> locale.setlocale(locale.LC_NUMERIC, 'fi_FI')
'fi_FI'
>>> format(Decimal('1000'), 'n')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/decimal.py", line 3632, in __format__
    spec = _parse_format_specifier(specifier, _localeconv=_localeconv)
  File "/usr/lib/python3.2/decimal.py", line 5628, in
_parse_format_specifier
    _localeconv = _locale.localeconv()
  File "/usr/lib/python3.2/locale.py", line 111, in localeconv
    d = _localeconv()
ValueError: Cannot convert byte to string

[skrah]

This fails in _localemodule.c: str2uni(). mbstowcs(NULL, s, 0) is
LC_CTYPE sensitive, but LC_CTYPE is UTF-8 in my terminal.

If I set LC_CTYPE and LC_NUMERIC together, things work.

This raises the question: If LC_CTYPE and LC_NUMERIC differ (and
since they are separate entities I assume they may differ), what
is the correct way to convert the separator and the decimal point?

a) call setlocale(LC_CTYPE, setlocale(LC_NUMERIC, NULL)) before
mbstowcs. This is not really an option.

b) use some kind of _mbstowcs_l
(http://msdn.microsoft.com/en-us/library/k1f9b8cy(VS.80).aspx), which
takes a locale parameter. But I don't
find such a thing on Linux.

[mdickinson]

I'm failing to reproduce this (with py3k) on OS X:

Python 3.2a0 (py3k:76866:76867, Dec 17 2009, 09:19:26) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> from decimal import *
>>> locale.setlocale(locale.LC_NUMERIC, 'fi_FI')
'fi_FI'
>>> format(Decimal('1000'), 'n')
'1.000'

The locale command, from the same Terminal prompt, gives me:

LANG="en_IE.UTF-8"
LC_COLLATE="en_IE.UTF-8"
LC_CTYPE="en_IE.UTF-8"
LC_MESSAGES="en_IE.UTF-8"
LC_MONETARY="en_IE.UTF-8"
LC_NUMERIC="en_IE.UTF-8"
LC_TIME="en_IE.UTF-8"
LC_ALL=

Just to be clear, is is true that you still get the same result without
involving Decimal at all? That is, am I correct in assuming that:

>>> import locale
>>> locale.setlocale(locale.LC_NUMERIC, 'fi_FI')
'fi_FI'
>>> locale.localeconv()

also gives you that ValueError?

[mdickinson]

What are the multibyte strings that mbstowcs is failing to convert?
On my machine, the separators come out as plain ASCII '.' (for thousands)
and ',' (for the decimal point).

[ericvsmith]

I can reproduce it on a Fedora (fc6) Linux box. It's not a decimal
problem, but a plain locale problem:

>>> import locale
>>> locale.setlocale(locale.LC_NUMERIC, 'fi_FI')
'fi_FI'
>>> locale.localeconv()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/python/py3k/Lib/locale.py", line 111, in localeconv
    d = _localeconv()
ValueError: Cannot convert byte to string
>>> 

Here's the contents of the struct lconv as returned by localeconv():

((gdb) p *l
$1 = {decimal_point = 0xb7b54020 ",", thousands_sep = 0xb7b54022 " ",
grouping = 0xb7b54024 "\003\003",
int_curr_symbol = 0x998858 "", currency_symbol = 0x998858 "",
mon_decimal_point = 0x998858 "", mon_thousands_sep = 0x998858 "",
mon_grouping = 0x998858 "", positive_sign = 0x998858 "", negative_sign
= 0x998858 "", int_frac_digits = 127 '\177',
frac_digits = 127 '\177', p_cs_precedes = 127 '\177', p_sep_by_space =
127 '\177', n_cs_precedes = 127 '\177',
n_sep_by_space = 127 '\177', p_sign_posn = 127 '\177', n_sign_posn =
127 '\177', int_p_cs_precedes = 127 '\177',
int_p_sep_by_space = 127 '\177', int_n_cs_precedes = 127 '\177',
int_n_sep_by_space = 127 '\177', int_p_sign_posn = 127 '\177',
int_n_sign_posn = 127 '\177'}

The problem is thousands_sep:
(gdb) p l->thousands_sep
$2 = 0xb7b54022 " "
(gdb) p (unsigned char)l->thousands_sep[0]
$3 = 160 ' '

[skrah]

Yes, it's a problem in _localemodule.c. This situation always
occurs when LC_NUMERIC is something like ISO8859-15, LC_CTYPE
is UTF-8 AND the decimal point or separator are in the range
128-255. Then mbstowcs tries to decode the character according
to LC_CTYPE and finds that the character is not valid UTF-8:

static PyObject*mbstowcs
str2uni(const char* s)
{
#ifdef HAVE_BROKEN_MBSTOWCS
    size_t needed = strlen(s);
#else
    size_t needed = mbstowcs(NULL, s, 0);
#endif

I can't see a portable way to fix this except:

block threads
set temporary LC_CTYPE
call mbstowcs
restore LC_CTYPE
unblock threads

I don't think this issue is important enough to do that. What
I do in cdecimal is raise an error "Invalid separator or
unsupported combination of LC_NUMERIC and LC_CTYPE".

[skrah]

Changed title (was: decimal.py: format failure with locale specifier)

[skrah]

I have a patch that fixes this specific issue. Probably there are similar
issues in other places, e.g. when LC_TIME and LC_CTYPE differ.

I suspect that this is related:

http://bugs.python.org/issue5905

[BreamoreBoy]

Could we have a patch review please. Also note that bpo-5905 has been closed.

[mcepl]

Is this the same as when tests with python-3.3.2 fails on me with RHEL-6?

test_locale (test.test_format.FormatTest) ... ERROR
test_non_ascii (test.test_format.FormatTest) ... test test_format failed
'\u20ac=%f' % (1.0,) =? '\u20ac=1.000000' ... yes
ok

======================================================================
ERROR: test_locale (test.test_format.FormatTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/matej/build/Extras/python3/Python-3.3.2/Lib/test/test_format.py", line 295, in test_locale
    localeconv = locale.localeconv()
  File "/home/matej/build/Extras/python3/Python-3.3.2/Lib/locale.py", line 111, in localeconv
    d = _localeconv()
UnicodeDecodeError: 'locale' codec can't decode byte 0xe2 in position 0: Invalid or incomplete multibyte or wide character

---

[skrah]

Matej Cepl <report@bugs.python.org> wrote:

Is this the same as when tests with python-3.3.2 fails on me with RHEL-6?

If LC_CTYPE is UTF-8 and LC_NUMERIC something like ISO-8859-2 then it's
the same issue.

[mcepl]

Hmm, so with this patch

diff -up Python-3.3.2/Lib/test/test_format.py.fixFormatTest Python-3.3.2/Lib/test/test_format.py
--- Python-3.3.2/Lib/test/test_format.py.fixFormatTest  2013-10-22 10:05:12.253426746 +0200
+++ Python-3.3.2/Lib/test/test_format.py        2013-10-22 10:16:58.510530570 +0200
@@ -288,7 +288,7 @@ class FormatTest(unittest.TestCase):
     def test_locale(self):
         try:
             oldloc = locale.setlocale(locale.LC_ALL)
-            locale.setlocale(locale.LC_ALL, '')
+            locale.setlocale(locale.LC_ALL, 'ps_AF')
         except locale.Error as err:
             self.skipTest("Cannot set locale: {}".format(err))
         try:

(or any other explicit locale, I have tried also en_IE) test doesn't fail.

Using Python-3.3.2 on RHEL-6 (kernel 2.6.32-358.23.2.el6.i686).

[mcepl]

To bpo-200912: now, system locale is UTF-8 all the way:
santiago:python3 (el6) $ locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC=en_IE.utf8
LC_TIME=en_IE.utf8
LC_COLLATE="en_US.utf8"
LC_MONETARY=en_IE.utf8
LC_MESSAGES="en_US.utf8"
LC_PAPER=en_IE.utf8
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT=en_IE.utf8
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=
santiago:python3 (el6) $