Add UTF-8 locale aliases

[serhiy-storchaka]

BPO	20076
Nosy	@malemburg, @loewis, @ezio-melotti, @serhiy-storchaka
Files	locale_utf8_aliases.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2014-10-01.21:30:38.102>
created_at = <Date 2013-12-26.23:29:08.866>
labels = ['type-feature', 'library']
title = 'Add UTF-8 locale aliases'
updated_at = <Date 2014-10-02.07:23:59.362>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2014-10-02.07:23:59.362>
actor = 'python-dev'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2014-10-01.21:30:38.102>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2013-12-26.23:29:08.866>
creator = 'serhiy.storchaka'
dependencies = []
files = ['33275']
hgrepos = []
issue_num = 20076
keywords = ['patch']
message_count = 5.0
messages = ['206974', '228066', '228120', '228123', '228170']
nosy_count = 5.0
nosy_names = ['lemburg', 'loewis', 'ezio.melotti', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20076'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[serhiy-storchaka]

The makelocalealias.py ignores UTF-8 mapping. Expected that this encoding is available for all locales. After enabling UTF-8 mapping in makelocalealias.py most aliases are optimized out except following:

+ 'be_bg.utf8': 'bg_BG.UTF-8',
+ 'c.utf8': 'en_US.UTF-8',
+ 'en_dl.utf8': 'en_DL.UTF-8',
+ 'en_zw.utf8': 'en_ZS.UTF-8',
+ 'ks_in@devanagari.utf8': 'ks_IN.UTF-8@devanagari',
+ 'pa_pk.utf8': 'pa_PK.UTF-8',
+ 'sd_in@devanagari.utf8': 'sd_IN.UTF-8@devanagari',
+ 'sr_yu.utf8': 'sr_RS.UTF-8',
+ 'sr_yu.utf8@cyrillic': 'sr_RS.UTF-8',
+ 'te_in.utf8': 'te_IN.UTF-8',
+ 'zh_sg.utf8': 'zh_SG.UTF-8',

Some of them maps to other country (en_zw.utf8 to en_ZS.UTF-8, sr_yu.utf8 to sr_RS.UTF-8) and these mappings are different from base mappings (en_zw to en_ZW.ISO8859-1, sr_yu to sr_RS.UTF-8@latin). The devanagari mappings just maps illformed locales. c.utf8 is yet one special case. Other mappings have no base entity without encoding.

Here is a patch which enables UTF-8 mapping in makelocalealias.py and adds all these mappings to locale alias table.

[malemburg]

Thanks, Serhiy

The patch looks good. Please apply.

[serhiy-storchaka]

After applying patch from bpo-20079, pa_pk.utf8, te_in.utf8, and zh_sg.utf8 are eliminated from this patch.

[python-dev]

New changeset c4284fa75ec7 by Serhiy Storchaka in branch '2.7':
Issue bpo-20076: Added non derived UTF-8 aliases to locale aliases table.
https://hg.python.org/cpython/rev/c4284fa75ec7

New changeset 3bff73df6dab by Serhiy Storchaka in branch '3.4':
Issue bpo-20076: Added non derived UTF-8 aliases to locale aliases table.
https://hg.python.org/cpython/rev/3bff73df6dab

New changeset 301b9a58021c by Serhiy Storchaka in branch 'default':
Issue bpo-20076: Added non derived UTF-8 aliases to locale aliases table.
https://hg.python.org/cpython/rev/301b9a58021c

[python-dev]

New changeset 840af1a073f7 by Serhiy Storchaka in branch '2.7':
Issue bpo-20076: Remove just added the sr_rs.utf8@latn alias because it is
https://hg.python.org/cpython/rev/840af1a073f7

New changeset ed92f06cdd8b by Serhiy Storchaka in branch '3.4':
Issue bpo-20076: Remove just added the sr_rs.utf8@latn alias because it is
https://hg.python.org/cpython/rev/ed92f06cdd8b

New changeset aa169e6d2e15 by Serhiy Storchaka in branch 'default':
Issue bpo-20076: Apply optimization in makelocalealias.py repeatedly.
https://hg.python.org/cpython/rev/aa169e6d2e15