Clearly document the use of PYTHONIOENCODING to set surrogateescape

[ncoghlan]

BPO	18713
Nosy	@malemburg, @ncoghlan, @pitrou, @vstinner, @benjaminp, @ezio-melotti, @abadger, @bitdancer, @serhiy-storchaka
Dependencies	bpo-18818: Empty PYTHONIOENCODING is not the same as nonexistent
Superseder	bpo-12832: The documentation for the print function should explain/point to how to control the sys.stdout encoding
Files	surrogateescape.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-12.15:19:33.999>
labels = ['type-feature', 'docs']
title = 'Clearly document the use of PYTHONIOENCODING to set surrogateescape'
updated_at = <Date 2017-03-24.16:29:59.624>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2017-03-24.16:29:59.624>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2013-08-12.15:19:33.999>
creator = 'ncoghlan'
dependencies = ['18818']
files = ['31414']
hgrepos = []
issue_num = 18713
keywords = ['patch']
message_count = 34.0
messages = ['194968', '194969', '194970', '195732', '195733', '195734', '195735', '195741', '195746', '195761', '195769', '195844', '195860', '195862', '195866', '195867', '195868', '195870', '195871', '195872', '195873', '195874', '195878', '195882', '195886', '195894', '195897', '195908', '195929', '195931', '195932', '195938', '195966', '242618']
nosy_count = 13.0
nosy_names = ['lemburg', 'ncoghlan', 'pitrou', 'vstinner', 'abadger1999', 'benjamin.peterson', 'ezio.melotti', 'a.badger', 'Arfrever', 'r.david.murray', 'nikratio', 'docs@python', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'needs patch'
status = 'pending'
superseder = '12832'
type = 'enhancement'
url = 'https://bugs.python.org/issue18713'
versions = ['Python 3.4']

[ncoghlan]

One problem with Unicode in 3.x is that surrogateescape isn't normally enabled on stdin and stdout. This means the following code will fail with UnicodeEncodeError in the presence of invalid filesystem metadata:

    print(os.listdir())

We don't really want to enable surrogateescape on sys.stdin or sys.stdout unilaterally, as it increases the chance of data corruption errors when the filesystem encoding and the IO encodings don't match.

Last night, Toshio and I thought of a possible solution: enable surrogateescape by default for sys.stdin and sys.stdout on non-Windows systems if (and only if) they're using the same codec as that returned by sys.getfilesystemencoding() (allowing for codec aliases rather than doing a simple string comparison)

This means that for full UTF-8 systems (which includes most modern Linux installations), roundtripping will be enabled by default between the standard streams and OS facing APIs, while systems where the encodings don't match will still fail noisily.

A more general alternative is also possible: default to errors='surrogatescape' for *any* text stream that uses the filesystem encoding. It's primarily the standard streams we're interested in fixing, though.

[bitdancer]

My gut reaction to this is that it feels dangerous. That doesn't mean my gut is right, I'm just reporting my reaction :)

[ncoghlan]

Everything about surrogateescape is dangerous - we're trying to work
around the presence of bad data by at least allowing it to be
tunnelled through Python code without corrupting it further :)

[abadger]

Nick and I had talked about this at a recent conference and came to it from different directions. On the one hand, Nick made the point that any encoding of surrogateescape'd text to bytes via a different encoding is corrupting the data as a whole. On the other hand, I made the point that raising an exception when doing something as basic as printing something that's text type was reintroducing the issues that python2 had wrt unicode, bytes, and encodings -- particularly with the exception being raised far from the source of the problem (when the data is introduced into the program).

After some thought, Nick came up with this solution. The idea is that surrogateescape was originally accepted to allow roundtripping data from the OS and back when the OS considers it to be a "string" but python does not consider it to be "text". When that's the case, we know what the encoding was used to attempt to construct the text in python. If that same encoding is used to re-encode the data on the way back to the OS, then we're successfully roundtripping the data we were given in the first place. So this is just applying the original goal to another API.

[ncoghlan]

Which reminds me: I'm curious what "ls" currently does for malformed
filenames. The aim of this change would be to get 'python -c "import os;
print(os.listdir())"' to do the best it can to work without losing data in
such a situation.

[vstinner]

On Linux, the locale encoding is usually UTF-8. If a filename cannot
be decoded from UTF-8, invalid bytes are escaped to the surrogate
range using the PEP-383. If I create a UTF-8 text file and I try to
write the filename into this text file, the Python UTF-8 encoder
raises an error.

IMO Python must raise an error here because I want to generate a valid
UTF-8 text file, not a text file only readable by Python if the locale
encoding is UTF-8.

So using surrogateescape error handler if the encoding is
sys.getfilesystemencoding() is *not* a good idea.

What is your use case where you need to display a filename? Is it
displayed to the terminal, into a file or in a graphical window? Why
not escaping surrogate just to format the filename, as Gnome does? See
for example:
https://developer.gnome.org/glib/2.34/glib-Character-Set-Conversion.html#g-filename-display-name

[vstinner]

2013/8/21 Nick Coghlan <report@bugs.python.org>:

Which reminds me: I'm curious what "ls" currently does for malformed
filenames. The aim of this change would be to get 'python -c "import os;
print(os.listdir())"' to do the best it can to work without losing data in
such a situation.

The "ls" command works on bytes, not on characters. You can
reimplement "ls" with:

• Unicode: os.listdir(str), os.fsencode() and sys.stdout.buffer
• bytes: os.listdir(bytes) and sys.stdout.buffer

os.fsencode() does exactly the opposite of os.fsdecode(). There is a
unit test to check that :-)

I ensured that all OS functions can be used directly with bytes
filenames in Python 3. That's why I added os.environb for example.

[ncoghlan]

Think sysadmins running scripts on Linux, writing to the console or a pipe.
I agree the generalisation is a bad idea, so only consider the original
proposal that was specifically limited to the standard streams.

Specifically, if a system is properly configured to use UTF-8 for all
interfaces, I shouldn't have to live in fear of Python steps in a command
pipeline falling over because it happens to encounter a filename encoded
with latin-1 (etc).

If the bytes oriented os tools like ls don't fall over on it, then neither
should Python. This is about treating the standard streams as OS
interfaces, as long as they're using the filesystem encoding.

[pitrou]

After some thought, Nick came up with this solution. The idea is that
surrogateescape was originally accepted to allow roundtripping data
from the OS and back when the OS considers it to be a "string" but
python does not consider it to be "text". When that's the case, we
know what the encoding was used to attempt to construct the text in
python. If that same encoding is used to re-encode the data on the
way back to the OS, then we're successfully roundtripping the data we
were given in the first place. So this is just applying the original
goal to another API.

I think that outlook is a bit naïve. The text source is not always the
same as the text destination, i.e. your surrogateescape-decoded data may
come from a database or some JSON API, so there's no reason to think
that the end of the stdout pipe will share the same convention.

I'm myself quite partial to the "round-tripping" use case, but I'm not
sure we can solve it as bluntly. If it's merely for printing out data,
maybe we can an os.fsescape() function to allow for representation of
broken filenames.

[bitdancer]

I think the essential use case is using a python program in a unix pipeline. I'm very sympathetic to that use case, despite my unease.

[vstinner]

Currently, Python 3 fails miserabily when it gets a non-ASCII
character from stdin or when it tries to write a byte encoded as a
Unicode surrogate to stdout.

It works fine when OS data can be decoded from and encoded to the
locale encoding. Example on Linux with UTF-8 data and UTF-8 locale
encoding:

$ mkdir test
$ cd test
$ touch héhé.txt
$ ls
héhé.txt
$ python3 -c 'import os; print(", ".join(os.listdir()))'
héhé.txt
$ echo "héhé"|python3 -c 'import sys; sys.stdout.write(sys.stdin.read())'|cat
héhé

It fails miserabily when OS data cannot be decoded from or encoded to
the locale encoding. Example on Linux with UTF-8 data and ASCII locale
encoding:

$ mkdir test
$ cd test
$ touch héhé.txt
$ export LANG=  # switch to ASCII locale encoding
$ ls
h??h??.txt
$ python3 -c 'import os; print(", ".join(os.listdir()))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position
1-2: ordinal not in range(128)

$ echo "héhé"|LANG= python3 -c 'import sys;
sys.stdout.write(sys.stdin.read())'|cat
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vstinner/prog/python/default/Lib/encodings/ascii.py",
line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
1: ordinal not in range(128)

The ls output is not the expected "héhé" string, but it is an issue
with the console output, not the ls program. ls does just write raw
bytes to stdout:

$ ls|hexdump -C
00000000  68 c3 a9 68 c3 a9 2e 74  78 74 0a                 |h..h...txt.|
0000000b

("héhé" encoded to UTF-8 gives b'h\xc3\xa9h\xc3\xa9')

I agree that we can do something to improve the situation on standard
streams, but only on standard streams. It is already possible to
workaround the issue by forcing the surrogateescape error handler on
stdout:

$ LANG= PYTHONIOENCODING=utf-8:surrogateescape python3 -c 'import os;
print(", ".join(os.listdir()))'
héhé.txt

Something similar can be done in Python. For example,
test.support.regrtest reopens sys.stdout to set the error handle to
"backslashreplace". Extract of the replace_stdout() function:

    sys.stdout = open(stdout.fileno(), 'w',
        encoding=sys.stdout.encoding,
        errors="backslashreplace",
        closefd=False,
        newline='\n')

[vstinner]

Attached patch changes the error handle of stdin, stdout and stderr to surrogateescape by default. It can still be changed explicitly using the PYTHONIOENCODING environment variable.

[serhiy-storchaka]

The surrogateescape error handler works only with UTF-8.

As a side effect of this change an input from stdin will be incompatible in general with extensions which implicitly encode a string to bytes with UTF-8 (e.g. tkinter, XML parsers, sqlite3, datetime, locale, curses, etc.)

[vstinner]

"The surrogateescape error handler works with any codec."

The surrogatepass only works with utf-8 if I remember correctly. The
surrogateescape error handler works with any codec, especially ascii.

"As a side effect of this change an input from stdin will be incompatible
in general with extensions which implicitly encode a string to bytes with
UTF-8 (e.g. tkinter, XML parsers, sqlite3, datetime, locale, curses, etc.)"

Correct, but it's not something new: os.listdir(), sys.argv, os.environ and
other functions using os.fsdecode(). Applications should already have to
support surrogates.