bugs in unicodedata.normalize: u1176, u11a7 and u11c3

[Pusnow]

BPO	29456
Nosy	@malemburg, @loewis, @vstinner, @ezio-melotti, @animalize, @zhangyangyu, @Pusnow, @miss-islington
PRs	bpo-29456: bugs in unicodedata.normalize: u1176, u11a7 and u11c3 #1958 [3.7] bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958) #7702 [3.6] bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958) #7703 [2.7] bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958) #7704
Files	u1176.patch u11a7u11c3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-15.13:28:49.407>
created_at = <Date 2017-02-06.04:27:52.177>
labels = ['3.8', 'type-bug', '3.7', 'expert-unicode']
title = 'bugs in unicodedata.normalize: u1176, u11a7 and u11c3'
updated_at = <Date 2018-06-18.14:21:55.246>
user = 'https://github.com/Pusnow'

bugs.python.org fields:

activity = <Date 2018-06-18.14:21:55.246>
actor = 'xiang.zhang'
assignee = 'none'
closed = True
closed_date = <Date 2018-06-15.13:28:49.407>
closer = 'xiang.zhang'
components = ['Unicode']
creation = <Date 2017-02-06.04:27:52.177>
creator = 'pusnow'
dependencies = []
files = ['46535', '46536']
hgrepos = []
issue_num = 29456
keywords = ['patch']
message_count = 23.0
messages = ['287077', '287078', '287079', '295123', '295171', '295172', '299214', '299657', '300039', '300046', '300576', '300933', '313056', '315214', '319591', '319608', '319609', '319610', '319615', '319701', '319719', '319802', '319886']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'vstinner', 'ezio.melotti', 'malin', 'xiang.zhang', 'pusnow', 'miss-islington']
pr_nums = ['1958', '7702', '7703', '7704']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29456'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[Pusnow]

unicodedata can't normalize(NFC) hangul strings which contain \u1176(HANGUL JUNGSEONG A-O).

>>> from unicodedata import normalize
>>> normalize("NFC", "\u1100\u1176\u11a8")
'깍'

=> should be "\u1100\u1176\u11a8" not '깍' (\uae4d)

I attached a patch for this issue. (Fixing boundary of modern medial vowels)

[zhangyangyu]

How about the third character's range? The code seems assuming it's [11a7..11c3] while the spec is [11a8..11c2]?

>>> unicodedata.normalize("NFC", "\u1100\u1175\u11a7")
'기'

while it should be '기ᆧ'?

[Pusnow]

I think you are right. The modern final consonants is [11a8..11c2].
I attached another patch for this issue.

[Pusnow]

Is there anything need more?

[zhangyangyu]

We have moved our code hosting to GitHub, would you mind turn your patch into a GitHub PR first Wonsup?

[Pusnow]

Ok, I'll do it.

[Pusnow]

Any updates? I need this fix for my project.

[Pusnow]

I added some test cases for this issue. Please, someone check this.

[Pusnow]

I think it can be merged. Is there anything I need to do?

[zhangyangyu]

Hi Wonsup, sorry for the delay. I get really busy with my work these days. If no one get involved I'd try to find time reviewing your patch this week.