Unicode problem with TZ

[theller]

BPO	1040
Nosy	@loewis, @theller, @amauryfa

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2007-08-30.14:40:27.467>
created_at = <Date 2007-08-28.06:05:53.228>
labels = ['OS-windows']
title = 'Unicode problem with TZ'
updated_at = <Date 2007-08-30.14:40:27.450>
user = 'https://github.com/theller'

bugs.python.org fields:

activity = <Date 2007-08-30.14:40:27.450>
actor = 'loewis'
assignee = 'loewis'
closed = True
closed_date = <Date 2007-08-30.14:40:27.467>
closer = 'loewis'
components = ['Windows']
creation = <Date 2007-08-28.06:05:53.228>
creator = 'theller'
dependencies = []
files = []
hgrepos = []
issue_num = 1040
keywords = []
message_count = 5.0
messages = ['55351', '55352', '55426', '55427', '55481']
nosy_count = 3.0
nosy_names = ['loewis', 'theller', 'amaury.forgeotdarc']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1040'
versions = ['Python 3.0']

[theller]

In my german version of winXP SP2, python3 cannot import the time module:

c:\svn\py3k\PCbuild>python_d
Python 3.0x (py3k:57600M, Aug 28 2007, 07:58:23) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 9-11:
invalid data
[36719 refs]
>>> ^Z

The problem is that the libc '_tzname' variable contains umlauts. For
comparison, here is what Python2.5 does:

c:\svn\py3k\PCbuild>\python25\python
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.tzname
('Westeurop\xe4ische Normalzeit', 'Westeurop\xe4ische Normalzeit')
>>>

[theller]

BTW, setting the environment variable TZ to, say, 'GMT' makes the
problem go away.

[amauryfa]

I have a patch for this, which uses MBCS conversion instead of relying
on the default utf-8 (here and several other places). Tested on a French
version of winXP.

Which leads me to the question: should Windows use MBCS encoding by
default when converting between char* and PyUnicode, and not utf-8?
There are some other tracker items which would benefit from this.

After all, C strings can only come from 1) python code, 2) system i/o
and messages, and 3) constants in source code.
IMO, 1) can use the representation it prefers, 2) would clearly lead to
less error if handled as MBCS and 3) only uses 7bit ascii.
There is very little need for utf-8 here.

[theller]

IMO the very best would be to avoid as many conversions as possible by
using the wide apis on Windows. Not for _tzname maybe, but for env
vars, sys.argv, sys.path, and so on. Not that I would have time to work
on that...

[loewis]

This is now fixed in r57720.

Using wide APIs would be possible through GetTimeZoneInformation,
however, then TZ won't be supported anymore (unless the CRT code to
parse TZ is duplicated).