Stricter codec names

[ezio-melotti]

BPO	5902
Nosy	@malemburg, @birkenfeld, @abalkin, @pitrou, @ezio-melotti
Files	iana.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/malemburg'
closed_at = <Date 2011-02-24.09:06:08.752>
created_at = <Date 2009-05-02.08:00:19.548>
labels = ['type-bug', 'library', 'docs']
title = 'Stricter codec names'
updated_at = <Date 2011-02-24.09:29:13.971>
user = 'https://github.com/ezio-melotti'

bugs.python.org fields:

activity = <Date 2011-02-24.09:29:13.971>
actor = 'lemburg'
assignee = 'lemburg'
closed = True
closed_date = <Date 2011-02-24.09:06:08.752>
closer = 'lemburg'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2009-05-02.08:00:19.548>
creator = 'ezio.melotti'
dependencies = []
files = ['20873']
hgrepos = []
issue_num = 5902
keywords = []
message_count = 15.0
messages = ['86933', '86935', '86937', '86956', '87034', '87103', '87140', '87144', '87226', '129238', '129239', '129248', '129254', '129255', '129257']
nosy_count = 6.0
nosy_names = ['lemburg', 'georg.brandl', 'belopolsky', 'pitrou', 'ezio.melotti', 'mrabarnett']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5902'
versions = ['Python 2.6', 'Python 3.0', 'Python 3.1', 'Python 2.7']

[ezio-melotti]

I noticed that codec names1:

1. can contain random/unnecessary spaces and punctuation;
2. have several aliases that could probably be removed;

A few examples of valid codec names (done with Python 3):
>>> s = 'xxx'
>>> s.encode('utf')
b'xxx'
>>> s.encode('utf-')
b'xxx'
>>> s.encode('}Utf~->8<-~siG{ ;)')
b'\xef\xbb\xbfxxx'

'utf' is an alias for UTF-8 and that doesn't quite make sense to me that
'utf' alone refers to UTF-8.
'utf-' could be a mistyped 'utf-8', 'utf-7' or even 'utf-16'; I'd like
it to raise an error instead.
The third example is not probably something that can be found in the
real world (I hope) but it shows how permissive the parsing of the names is.

Apparently the whitespaces are removed and the punctuation is used to
split the name in several parts and then the check is performed.

About the aliases: in the documentation the "official" name for the
UTF-8 codec is 'utf_8' and there are 3 more aliases: U8, UTF, utf8. For
ISO-8859-1, the "official" name is 'latin_1' and there are 7 more
aliases: iso-8859-1, iso8859-1, 8859, cp819, latin, latin1, L1.
The Zen says "There should be one—and preferably only one—obvious way to
do it.", so I suggest to

1. disallow random punctuation and spaces within the name (only allow
   leading and trailing spaces);
2. change the default names to, for example: 'utf-8', 'iso-8859-1'
   instead of 'utf_8' and 'iso8859_1'. The name are case-insentive.
3. remove the unnecessary aliases, for example: 'UTF', 'U8' for UTF-8
   and 'iso8859-1', '8859', 'latin', 'L1' for ISO-8859-1;

This last point could break some code and may need some
DeprecationWarning. If there are good reason to keep around these codecs
only the other two issues can be addressed.
If the name of the codec has to be a valid variable name (that is,
without '-'), only the documentation could be changed to have 'utf-8',
'iso-8859-1', etc. as preferred name.

http://docs.python.org/3.0/library/codecs.html#standard-encodings

[birkenfeld]

I don't think this is a good idea. Accepting all common forms for
encoding names means that you can usually give Python an encoding name
from, e.g. a HTML page, or any other file or system that specifies an
encoding. If we only supported, e.g., "UTF-8" and no other spelling,
that would make life much more difficult. If you look into
encodings/init.py, you can see that throwing out all
non-alphanumerics is a conscious design choice in encoding name
normalization.

The only thing I don't know is why "utf" is an alias for utf-8.

Assigning to Marc-Andre, who implemented most of codecs.

[pitrou]

Is there any reason for allowing "utf" as an alias to utf-8? It sounds
much too ambiguous. The other silly variants (those with lots of
spurious puncutuations characters) could be forbidden too.

[mrabarnett]

How about a 'full' form and a 'key' form generated by the function:

def codec_key(name):
    return name.lower().replace("-", "").replace("_", "")

The key form would be the key to an available codec, and the key
generated by a user-supplied codec name would have to match one of those
keys.

For example:

Full: "UTF-8", key: "utf8".

Full: "ISO-8859-1", key: "iso88591".

[ezio-melotti]

Actually I'd like to have some kind of convention mainly when the user
writes the encoding as a string, e.g. s.encode('utf-8'). Indeed, if the
encoding comes from a webpage or somewhere else it makes sense to have
some flexibility.

I think that 'utf-8' is the most widely used name for the UTF-8 codec
and it's not even mentioned in the table of the standard encodings. So
someone will use 'utf-8', someone else 'utf_8' and some users could even
pick one of the aliases, like 'U8'.

Probably is enough to add 'utf-8', 'iso-8859-1' and similar as
"preferred form" and explain why and how the codec names are normalized
and what are the valid aliases.

Regarding the ambiguity of 'UTF', it is not the only one, there's also
'LATIN' among the aliases of ISO-8859-1.

[malemburg]

On 2009-05-02 11:20, Georg Brandl wrote:

Georg Brandl <georg@python.org> added the comment:

I don't think this is a good idea. Accepting all common forms for
encoding names means that you can usually give Python an encoding name
from, e.g. a HTML page, or any other file or system that specifies an
encoding. If we only supported, e.g., "UTF-8" and no other spelling,
that would make life much more difficult. If you look into
encodings/init.py, you can see that throwing out all
non-alphanumerics is a conscious design choice in encoding name
normalization.

The only thing I don't know is why "utf" is an alias for utf-8.

Assigning to Marc-Andre, who implemented most of codecs.

-1 on making codec names strict.

The reason why we have to many aliases is to enhance compatibility
with other software and data, not to encourage use of these aliases
in Python itself.

[birkenfeld]

So, do you also think "utf" and "latin" should stay?

[mrabarnett]

Well, there are multiple UTF encodings, so no to "utf".

Are there multiple Latin encodings? Not in Python 2.6.2 under those names.

I'd probably insist on names that are strictish(?), ie correct, give or
take a '-' or '_'.

[malemburg]

On 2009-05-04 19:04, Georg Brandl wrote:

Georg Brandl <georg@python.org> added the comment:

So, do you also think "utf" and "latin" should stay?

For Python 3.x, I think those can be removed. For 2.x it's better to
keep them.

Note that UTF-8 was the first official Unicode transfer encoding,
that's why it's sometimes referred to as "UTF".

The situation is similar for Latin-1. It was the first of a series of
encodings defined by ECMA which was later published by ISO under the name
ISO-8859 - long after the name "Latin-1" became popular which is why
it's the default name in Python.