Clearly explain the bare minimum Python 3 users should know about Unicode

[ncoghlan]

BPO	13997
Nosy	@terryjreedy, @pfmoore, @ncoghlan, @pitrou, @vstinner, @giampaolo, @ezio-melotti, @merwok, @florentx, @JimJJewett
Superseder	bpo-4153: Unicode HOWTO up to date?

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-01-28.02:20:15.458>
created_at = <Date 2012-02-12.04:33:06.630>
labels = ['type-feature', 'expert-unicode', 'docs']
title = 'Clearly explain the bare minimum Python 3 users should know about Unicode'
updated_at = <Date 2013-01-28.02:20:15.457>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2013-01-28.02:20:15.457>
actor = 'ezio.melotti'
assignee = 'docs@python'
closed = True
closed_date = <Date 2013-01-28.02:20:15.458>
closer = 'ezio.melotti'
components = ['Documentation', 'Unicode']
creation = <Date 2012-02-12.04:33:06.630>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 13997
keywords = []
message_count = 27.0
messages = ['153164', '153167', '153169', '153170', '153171', '153179', '153182', '153183', '153189', '153190', '153191', '153198', '153202', '153206', '153360', '153606', '153612', '153645', '153653', '157210', '180728', '180743', '180775', '180793', '180795', '180819', '180821']
nosy_count = 14.0
nosy_names = ['terry.reedy', 'paul.moore', 'ncoghlan', 'pitrou', 'vstinner', 'giampaolo.rodola', 'nadeem.vawda', 'ezio.melotti', 'eric.araujo', 'cvrebert', 'flox', 'docs@python', 'tshepang', 'Jim.Jewett']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '4153'
type = 'enhancement'
url = 'https://bugs.python.org/issue13997'
versions = ['Python 3.2', 'Python 3.3', 'Python 3.4']

[ncoghlan]

(This proposes a new builtin, so may need to become a PEP)

A common programming task is "I want to process this text file, I know it's in an ASCII compatible encoding, I don't know which one specifically, but I'm only manipulating the ASCII parts so it doesn't matter".

In Python 2, you handle that task by doing:

    f = open(fname)

The non-ASCII parts are then carried along as 8-bit bytes and reproduced faithfully when written back out.

In Python 3, you handle it by doing:

    f = open(fname, encoding="ascii", errors="surrogateescape")

The non-ASCII parts are then carried along as code points in the Unicode Private Use Area and reproduced faithfully when written back out.

It would be significantly more friendly to beginners (and migrants from Python 2) if the following shorthand spelling was available out of the box:

    f = open_ascii(fname)

[elibendersky]

Would not adding a new keyword arg to open() be less intrusive and more consistent?

I.e. open(fname, asciionly=True) or something similar.

[pitrou]

Hmm, what happened to "not every one-liner should be a builtin function"?

I'm only manipulating the ASCII parts

How can you be sure about that?

It would be significantly more friendly to beginners (and migrants from
Python 2)

To the point that many of them would stop thinking about the problem, and start producing unicode-incompatible code.
The idea that it may be presented as a porting recipe is IMO a good reason to oppose introducing this new function.

[cvrebert]

@bendersky:

Unlike open()'s other arguments, that one wouldn't be orthogonal though. It would be possible to write e.g.:

f = open(fname, encoding="big5", errors="replace", ascii_only=True)

which seems disturbing, IMO. It would be nicer to rule out such impossible combinations categorically.

[ncoghlan]

No point to adding a new keyword arg - if people are going to do something like that, they may as well learn to use "errors" and "encoding" properly.

Adding open_ascii() would be an acknowledgement that "basically ASCII, but maybe with a few other bytes that just need to round-trip correctly" is a common enough use case to special case (in particular, it's convenient to have algorithms than can operate on both utf-8 and all 8-bit extended ASCII variants, including latin-1).

The downside to using surrogateescape is that if you ever *do* feed it a file in a non-ASCII compatible encoding and then perform ASCII-based manipulations, you'll get mojibake instead of an early UnicodeDecodeError. (i.e. exactly the same problem this kind of thing can cause in Python 2)

[merwok]

IMO it is a fact that the characters used by human languages are stored as bytes by computers, so a programmer needs to know the basics about text handling and encoding. I don’t like the idea of a built-in function helping people to put their hands on their ears and sing “la-la-la everything is ASCII”. I know that it is hard to find oneself confronted with a UnicodeDecodeError, I’ve been there, but then I learned. My hope is that good explanations in the FAQ, howto and library ref (with good PageRank, so that people googling error messages find us) can help people understand how to work with text.

That said, I’m going to re-read Armin’s post about Python 3 and Nick’s reply to it to understand your position better.

can’t-put-my-damn-name-in-so-many-websites-in-2012-grumble’ly yours

[ncoghlan]

Pondering it further (and reading subsequent comments here and in the thread), I agree an open_ascii() builtin would be a step backwards, not forwards.

So, morphing this issue into a documentation one to work out:

• the bare minimum we think Python 3 users should be learning about Unicode
• deciding where to document that (with a reference to the Unicode HOWTO for anyone that wants to know more)

Some ideas specifically in the context of text files (for readers already familiar with the basic concept of text encodings):

1. The world is moving towards standardising on UTF-8 as the binary encoding used to store text files. However, we're a long way from living in that world right now. Other encodings (many, but far from all, ASCII compatible) will be encountered quite often, either as the default encoding on a particular platform, or as the encoding of a particular text file. Dealing with these correctly requires additional work.

2. To maximise the chance of correct local interoperability, Python 3's default choice of encoding is actually taken from the underlying platform rather than being forced to UTF-8. While it is becoming more and more common for platforms to set their preferred encoding to UTF-8, this is not yet universal (notably, Windows still does not use UTF-8 as the default encoding for text files in order to preserve compatibility with various Unicode-unaware legacy applications).

To handle this correctly in cross-platform applications and libraries, it is often necessary to explicitly pass "encoding='utf-8'" when opening a UTF-8 encoded text file.

The default encoding on a given platform can be checked by running "import locale; locale.getpreferredencoding()" at the interactive prompt.

3. Currently, it is still fairly common to encounter text files that are known to be stored in an ASCII-compatible text encoding without knowing precisely *which* encoding is used. The Python 2 text model allowed such files to be processed naively simply by assuming they were in an ASCII-compatible encoding and passing any non-ASCII characters faithfully through to the result. This permissive behaviour can be requested explicitly in Python 3 by passing "encoding='ascii'" and "errors='surrogateescape'" when opening a text file.

This approach parallels the behaviour of Python 2 and works correctly so long as it is fed data solely in ASCII compatible encodings (such as UTF-8 and latin-1). Behaviour when fed data that uses other encodings is unpredictable - common symptoms include Unicode encoding and decoding errors at unexpected points in a program, as well as silent corruption of the output text.

[elibendersky]

If the concept is accepted. I see no better place for this than the
Unicode HOWTO. If it's too long, then a TL;DR; section should be added
in the beginning detailing "the bare minimum". No need to scatter such
information in bits and pieces around the documentation. That's what
the Unicode HOWTO is for.

[vstinner]

A common programming task is "I want to process this text file,
I know it's in an ASCII compatible encoding, I don't know which
one specifically, but I'm only manipulating the ASCII parts
so it doesn't matter".

Can you give more detail about this use case? Why would you ignore non-ASCII characters?

[ncoghlan]

Usually because the file may contain certain ASCII markers (or you're inserting such markers), but beyond that, you only care that it's in a consistent ASCII compatible encoding.

Parsing log files from sources that aren't set up correctly often falls into this category - you know the markers are ASCII, but the actual message contents may not be properly encoded. (e.g. they use a locale dependent encoding, but not all the log files are from the same machine and not all machines have their locale set up properly). (although errors="replace" can be a better option for such "read-only" use cases).

A use case where you really do need "errors='surrogateescape'" is when you're reformatting a log file and you want to preserve the encoding for the messages while manipulating the pure ASCII timestamps and message headers. In that case, surrogateescape is the right answer, because you can manipulate the ASCII bits freely while preserving the log message contents when you write the reformatted files back out. The reformatting script offers an API that says "put any ASCII compatible encoding in, and you'll get that same encoding back out".

You'll get weird behaviour (i.e. as you do in Python 2) if the assumption of an ASCII compatible encoding is ever violated, but that would be equally true if the script tried to process things at the raw bytes level.

The assumption of an ASCII compatibile text encoding is a useful one a lot of the time. The problem with Python 2 is it makes that assumption implicitly, and makes it almost impossible to disable it. Python 3, on the other hand, assumes very little by default (basically what it returns from sys.getfilesystemencoding() and locale.getpreferredencoding()), this requiring that the programmer know how to state their assumptions explicitly.