CVE-2019-9636: urlsplit does not handle NFKC normalization

[zooba]

BPO	36216
Nosy	@vstinner, @benjaminp, @jkloth, @ned-deily, @mcepl, @ezio-melotti, @vadmium, @koobs, @zooba, @tirkarthi
PRs	bpo-36216: Add check for characters in netloc that normalize to separators #12201 [3.7] bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) #12213 [3.6] bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) #12215 [2.7] bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) #12216 [3.5] bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) #12223 [3.4] bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) #12224 [2.7] bpo-36216: Only print test messages when verbose (GH-12291) #12291

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-03-12.21:25:47.134>
created_at = <Date 2019-03-06.17:37:20.433>
labels = ['type-security', '3.7', '3.8', 'expert-unicode']
title = 'CVE-2019-9636: urlsplit does not handle NFKC normalization'
updated_at = <Date 2021-05-11.14:28:00.197>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2021-05-11.14:28:00.197>
actor = 'larry'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-03-12.21:25:47.134>
closer = 'steve.dower'
components = ['Unicode']
creation = <Date 2019-03-06.17:37:20.433>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 36216
keywords = ['patch', 'security_issue']
message_count = 22.0
messages = ['337336', '337383', '337398', '337411', '337412', '337532', '337566', '337645', '337646', '337704', '337711', '337720', '337725', '337753', '337771', '337773', '337806', '337811', '339391', '339428', '339433', '393450']
nosy_count = 11.0
nosy_names = ['vstinner', 'benjamin.peterson', 'jkloth', 'ned.deily', 'mcepl', 'ezio.melotti', 'jeremy.kloth', 'martin.panter', 'koobs', 'steve.dower', 'xtreak']
pr_nums = ['12201', '12213', '12215', '12216', '12223', '12224', '12291']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue36216'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[zooba]

URLs encoded with Punycode/IDNA use NFKC normalization to decompose characters 1. This can result in some characters introducing new segments into a URL.

For example, \uFF03 is not equal to '#' under direct comparison, but normalizes to '#' which changes the fragment part of the URL. Similarly \u2100 normalizes to 'a/c' which introduces a path segment.

Currently, urlsplit() does not normalize, which may result in it returning a different netloc from what a browser would

>>> u = "https://example.com\uFF03@bing.com"
>>> urlsplit(u).netloc.rpartition("@")[2]
bing.com

>>> # Simulate
>>> u = "https://example.com\uFF03@bing.com".encode("idna").decode("ascii")
>>> urlsplit(u).netloc.rpartition("@")[2]
example.com

(Note that .netloc includes user/pass and .rpartition("@") is often used to remove it.)

This may be used to steal cookies or authentication data from applications that use the netloc to cache or retrieve this information.

The preferred fix for the urllib module is to detect and raise ValueError if NFKC-normalization of the netloc introduce any of '/?#@:'. Applications that want to avoid this error should perform their own decomposition using unicodedata or transcode to ASCII via IDNA.

>>> # New behavior
>>> u = "https://example.com\uFF03@bing.com"
>>> urlsplit(u)
...
ValueError: netloc 'example.com#@bing.com' contains invalid characters under NFKC normalization

>>> # Workaround 1
>>> u2 = unicodedata.normalize("NFKC", u)
>>> urlsplit(u2)
SplitResult(scheme='https', netloc='example.com', path='', query='', fragment='@bing.com')

>>> # Workaround 2
>>> u3 = u.encode("idna").decode("ascii")
>>> urlsplit(u3)
SplitResult(scheme='https', netloc='example.com', path='', query='', fragment='@bing.com')

Note that we do not address other characters, such as those that convert into period. The error is only raised for changes that affect how urlsplit() locates the netloc and the very common next step of removing credentials from the netloc.

This vulnerability was reported by Jonathan Birch of Microsoft Corporation and Panayiotis Panayiotou (p.panayiotou2@gmail.com) via the Python Security Response Team. A CVE number has been requested.

[vstinner]

FYI I added this vulnerability to:
https://python-security.readthedocs.io/vuln/urlsplit-nfkc-normalization.html

[zooba]

New changeset 16e6f7d by Steve Dower in branch 'master':
bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201)
16e6f7d

[zooba]

New changeset daad2c4 by Steve Dower in branch '3.7':
bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201)
daad2c4

[zooba]

New changeset e37ef41 by Steve Dower in branch '2.7':
bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201)
e37ef41

[zooba]

This issue is now assigned CVE-2019-9636

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9636

[jkloth]

A missed print statement in the 2.7 patch is causing the tests to fail.

Line 647 of Lib/test/test_urlparse.py

[larryhastings]

New changeset 62d3654 by larryhastings (Steve Dower) in branch '3.4':
bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) (bpo-12224)
62d3654

[larryhastings]

New changeset c0d9511 by larryhastings (Steve Dower) in branch '3.5':
bpo-36216: Add check for characters in netloc that normalize to separators (GH-12201) (bpo-12223)
c0d9511

[zooba]

A missed print statement in the 2.7 patch is causing the tests to fail.

How does that cause tests to fail? Is it going to stderr? Or just causing an error.