CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@

[hokousya]

BPO	36742
Nosy	@orsenthil, @vstinner, @larryhastings, @benjaminp, @ned-deily, @ezio-melotti, @ambv, @zooba, @stratakis, @miss-islington, @tirkarthi, @ret2libc
PRs	bpo-36742: Fixes handling of pre-normalization characters in urlsplit() #13017 [3.7] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13023 [3.6] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13024 [2.7] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13025 [3.5] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13042 bpo-36742: Corrects fix to handle decomposition in usernames #13812 [3.7] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) #13813 [3.6] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) #13814 [2.7] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) #13815 [2.7] bpo-36742: Fix urlparse.urlsplit() error message for Unicode URL #13937 [3.5] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) (GH-13814) #14772

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-05-01.15:04:11.989>
created_at = <Date 2019-04-27.12:30:16.902>
labels = ['type-security', '3.7', '3.8', 'expert-unicode', 'release-blocker']
title = 'CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@'
updated_at = <Date 2019-09-07.06:33:27.509>
user = 'https://bugs.python.org/hokousya'

bugs.python.org fields:

activity = <Date 2019-09-07.06:33:27.509>
actor = 'larry'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-05-01.15:04:11.989>
closer = 'steve.dower'
components = ['Unicode']
creation = <Date 2019-04-27.12:30:16.902>
creator = 'hokousya'
dependencies = []
files = []
hgrepos = []
issue_num = 36742
keywords = ['patch', '3.5regression', '3.6regression', '3.7regression']
message_count = 23.0
messages = ['340983', '341006', '341092', '341125', '341150', '341151', '341171', '341206', '341207', '341208', '341212', '341282', '344595', '344596', '344597', '344601', '344623', '344973', '344981', '345116', '345218', '347880', '351285']
nosy_count = 13.0
nosy_names = ['orsenthil', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'lukasz.langa', 'steve.dower', 'cstratak', 'miss-islington', 'xtreak', 'hokousya', 'rschiron']
pr_nums = ['13017', '13023', '13024', '13025', '13042', '13812', '13813', '13814', '13815', '13937', '14772']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue36742'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[hokousya]

urllib.parse.urlsplit raises an exception for an url including a non-ascii hostname in NFKD form and a port number.

example:
>>> urlsplit('http://\u30d5\u309a:80')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ito/.maltybrew/deen/lib/python3.7/urllib/parse.py", line 437, in urlsplit
    _checknetloc(netloc)
  File "/Users/ito/.maltybrew/deen/lib/python3.7/urllib/parse.py", line 407, in _checknetloc
    "characters under NFKC normalization")
ValueError: netloc 'プ:80' contains invalid characters under NFKC normalization
>>> urlsplit('http://\u30d5\u309a')
SplitResult(scheme='http', netloc='プ', path='', query='', fragment='')
>>> urlsplit(unicodedata.normalize('NFKC', 'http://\u30d5\u309a:80'))
SplitResult(scheme='http', netloc='プ:80', path='', query='', fragment='')

I believe this behavior was introduced at Python 3.7.3. Python 3.7.2 doesn't raise any exception for these lines.

[tirkarthi]

This could be due to bpo-36216.

[zooba]

Yes, it's due to that. I guess we need to do netloc.rpartition(':') like we currently do for '@' in _checknetloc.

Promoting to release blocker and security issue to match the original issue. I can't get to this today, but I should be able to at the PyCon sprints next week if nobody else gets it sooner.

[zooba]

I found the time to get the first patch. Hopefully backports to 3.6 and 3.7 are easy, but I think 2.7 will take manual steps.

Chihiro Ito - if you have other test scenarios, it would be great if you could try them out with the fix in PR 13017. It should be easy enough to copy into your installed Python.

[zooba]

New changeset d537ab0 by Steve Dower in branch 'master':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
d537ab0

[miss-islington]

New changeset 4d723e7 by Miss Islington (bot) in branch '3.7':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
4d723e7

[hokousya]

I have confirmed that all of my app's test cases have passed.

What I've done:

1. Installed Python 3.7.3.
2. Replaced urllib/parse.py with the one from 781ffb1.
3. Ran my app's test cases.

Thank you for the quick fix!