Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale

[vstinner]

BPO	19977
Nosy	@loewis, @atsuoishimoto, @ncoghlan, @pitrou, @vstinner, @larryhastings, @jwilk, @ezio-melotti, @abadger, @bitdancer, @vadmium, @serhiy-storchaka
Files	c_locale_surrogateescape.patch test_ls.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-04-27.23:57:39.901>
created_at = <Date 2013-12-13.16:40:20.152>
labels = ['type-bug', 'expert-unicode']
title = 'Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale'
updated_at = <Date 2017-12-18.14:31:51.323>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-12-18.14:31:51.323>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-27.23:57:39.901>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2013-12-13.16:40:20.152>
creator = 'vstinner'
dependencies = []
files = ['33122', '33124']
hgrepos = []
issue_num = 19977
keywords = ['patch']
message_count = 38.0
messages = ['206111', '206114', '206121', '206122', '206123', '206124', '206131', '206141', '206148', '206168', '207210', '207296', '207414', '210990', '213922', '213925', '213927', '213932', '213933', '215029', '215786', '215812', '215815', '217300', '217314', '217315', '217317', '217329', '217332', '217355', '217385', '217386', '217387', '284849', '284863', '284873', '284875', '308562']
nosy_count = 15.0
nosy_names = ['loewis', 'ishimoto', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'jwilk', 'ezio.melotti', 'a.badger', 'r.david.murray', 'Sworddragon', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'bkabrda']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19977'
versions = ['Python 3.5']

[vstinner]

When LANG=C is used to get the english language (which is a mistake, LC_CTYPE=C should be used instead) or when Python is started with an empty environment (no environment variable), Python gets the POSIX locale (aka "C locale") for the LC_CTYPE (encoding) locale.

Standard streams use the locale encoding, which is usually ASCII with POSIX locale on most platforms (except on AIX: ISO 8859-1). In this case, data read from the OS (environment variables, command line arguments, filenames, etc.) may contain surrogate characters because of the internal usage of the surrogateescape error handler (see the PEP-383 for the rationale).

The problem is that standard output uses the strict error handler, and so print() fails to display OS data like filenames.

Example, "ls" command in Python:
---

import os
for name in sorted(os.listdir()): print(name)

---

Try it with "LANG=C python ls.py" in a directory containing non-ASCII characters and you will get unicode errors.

Issues bpo-19846 and bpo-19847 are examples of this annoyance.

I propose to use also the surrogateescape error handler for sys.stdout if the POSIX locale is used for LC_CTYPE at startup. Attached patch implements this idea.

With the patch, "LANG=C python ls.py" almost works as filenames and stdout are byte streams, even if the Unicode type is used.

[vstinner]

Oh, in fact, sys.stdin is also modified by the patch (as I expected).

[Sworddragon]

What would happen if we call this example script with LANG=C on the patch?:

---

import os
for name in sorted(os.listdir('ä')):
	print(name)

---

Would it throw an exception on os.listdir('ä')?

[vstinner]

test_ls.py: test script producing invalid filenames and then trying to display them into stdout.

Output with UTF-8 locale, UTF-8 terminal and Python 3.3 (or unpatched 3.4, it's the same):

ascii.txt
<UnicodeError 'invalid_utf8:\udcff.txt'>
<UnicodeError 'latin1:\udce9.txt'>
utf8:é€.txt

Output with C locale (ASCII), UTF-8 terminal and Python 3.3:

ascii.txt
<UnicodeError 'invalid_utf8:\udcff.txt'>
<UnicodeError 'latin1:\udce9.txt'>
<UnicodeError 'utf8:\udcc3\udca9\udce2\udc82\udcac.txt'>

Output with C locale (ASCII), UTF-8 terminal and patched Python 3.4:

ascii.txt
invalid_utf8:�.txt
latin1:�.txt
utf8:é€.txt

You get no Unicode error with LANG=C, but you get mojibake instead.

[vstinner]

os.fsencode(text) always fail if text cannot be encoded to sys.getfilesystemencoding(). surrogateescape doesn't help here.

Your example is "artificial", you should not get 'ä'. All OS data is decoded from the filesystem encoding using the surrogateescape error handler (except on Windows, where strict is used, but it's a different story, Python uses Unicode functions when available so don't worry). So all these data can always be encoded back to bytes using os.fsencode().

More generally, os.fsencode(os.fsdecode(read_data)) == read_data is always true on Unix, with any filesystem (locale) encoding.

You may get Unicode data from other sources like files or a GUI, but I don't see what can be done here.

[pitrou]

When LANG=C is used to get the english language (which is a mistake,
LC_CTYPE=C should be used instead)

I think you mean LC_MESSAGES=C here.
(but it's not only about the English language; it's also about other locale parameters such as number formatting)

I think we should start thinking about making utf-8 the default filesystem encoding in 3.5 (under Unix).

[bitdancer]

Reintroducing moji-bake intentionally doesn't sound like a particularly good idea, wasn't that what python3 was supposed to help prevent?

It does seem like a utf-8 default is the Way of the Future. Or even the present, most places.

[abadger]

My impression was that python3 was supposed to help get rid of UnicodeError tracebacks, not mojibake. If mojibake was the problem then we should never have gone down the surrogateescape path for input.

[serhiy-storchaka]

Mojibake in input can cause decoding error in other application which consumes output of Python script. In some cases this can be even worse thin UnicodeError in producer.

But for C locale this makes sense. I think we should try this experiment in 3.5. There will be much time for testing before 3.5 beta 1.

[ncoghlan]

Getting rid of mojibake was the goal, surrogateescape was about dealing with cases where the "avoid mojibake" checks were spuriously breaking round-tripping between OS APIs due to other configuration errors (with LANG=C being set, or LANG not being set at all being the main problem). Other "high mojibake risk" power tools (like changing the encoding of an already open stream) are likely to return in the future, since there *are* cases where they're the right answer (e.g. you can't right an iconv equivalent in Python 3 at the moment, we need bpo-15216 implemented before that will be possible).

+1 for this solution - see bpo-19846 for the long discussion which got us to this point (there are a few unrelated tangents, a couple of them my fault, but this is definitely an improvement over the status quo.

[ncoghlan]

Larry: I'm assuming it's way too late to make a change like this for the 3.4 release?

Slavek: assuming this change is made for 3.5 upstream, we may want to look at backporting it as a 3.4 patch in Fedora (as part of the Python-3-by-default project). Otherwise it's very easy to provoke Python 3 into throwing Unicode errors when attempting to print data provided by the OS.

[larryhastings]

Yeah, unless there was a *huge* amount of support for changing this, it's way too late for 3.4.