SSL match_hostname fails for internationalized domain names

[abracadaber]

BPO	28414
Nosy	@tiran, @ned-deily, @alex, @njsmith, @1st1, @dstufft, @wumpus, @abracadaber, @kedare, @tialaramex
PRs	[bpo-28414] In SSL module, store server_hostname as an A-label #3010 [bpo-28414] Make all hostnames in SSL module IDN A-labels #5128 bpo-28414: ssl module idna test #5395 [3.7] [bpo-28414] Make all hostnames in SSL module IDN A-labels (GH-5128) #5843

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2018-02-24.06:08:31.831>
created_at = <Date 2016-10-11.08:02:36.045>
labels = ['expert-SSL', 'deferred-blocker', '3.7', '3.8', 'type-feature', 'expert-asyncio']
title = 'SSL match_hostname fails for internationalized domain names'
updated_at = <Date 2018-02-24.06:08:31.830>
user = 'https://github.com/abracadaber'

bugs.python.org fields:

activity = <Date 2018-02-24.06:08:31.830>
actor = 'njs'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2018-02-24.06:08:31.831>
closer = 'njs'
components = ['asyncio', 'SSL']
creation = <Date 2016-10-11.08:02:36.045>
creator = 'abracadaber'
dependencies = []
files = []
hgrepos = []
issue_num = 28414
keywords = ['patch']
message_count = 21.0
messages = ['278466', '278483', '278488', '278519', '279165', '279920', '292025', '295384', '295391', '295775', '298583', '299812', '299813', '299846', '306450', '310992', '311128', '311130', '312476', '312685', '312696']
nosy_count = 12.0
nosy_names = ['janssen', 'christian.heimes', 'ned.deily', 'alex', 'njs', 'yselivanov', 'dstufft', 'wumpus', 'abracadaber', 'Socob', 'kedare', 'tialaramex']
pr_nums = ['3010', '5128', '5395', '5843']
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28414'
versions = ['Python 3.7', 'Python 3.8']

[abracadaber]

In accordance with http://tools.ietf.org/html/rfc6125#section-6.4.2:
"If the DNS domain name portion of a reference identifier is an internationalized domain name, then an implementation MUST convert any U-labels [IDNA-DEFS] in the domain name to A-labels before checking the domain name."
The question is: Where in python stdlib should it to convert domain name from U-label to A-label? Should it be in ssl._dnsname_match, e.g.:
...
hostname = hostname.encode('idna').decode('utf-8')
...
Or should it be at ssl._dnsname_match caller level?

I found that error appears after using ssl.SSLContext.wrap_bio, which in turn uses internal newPySSLSocket, which in turn always decode server_hostname through:
PySSLSocket *self;
...
PyObject *hostname = PyUnicode_Decode(server_hostname, strlen(server_hostname), "idna", "strict");
...
self->server_hostname = hostname;
In this way, SSLSocket always contains U-label in its server_hostname field, and ssl._dnsname_match falis with "ssl.CertificateError: hostname ... doesn't match either of ..."

And i don't understand where is a bug, or is it a bug.

[tiran]

Thanks for bringing this to my attention. I can confirm that the code is broken. Further more there are no tests for IDN for server_hostname.

• server_hostname must be an IDN U-label (locälhost)
• SSL handshake correctly converts and sends TLS SNI as IDN A-label (xn--loclhost-2za)
• getpeercert() returns DNS SAN as IDN A-label. It's less than ideal but required.
• the serverhostname_callback is called with IDN U-label
• match_hostname() is called with IDN U-label

The bug is clearly in match_hostname(). The function fails to convert the hostname U-label to A-label before it compares the certificate.

I have a rough draft of a patch here https://github.com/tiran/cpython/tree/issue28414_idna_verify

By the way IDNA support in Python is broken in general, bpo-17305. We still don't support the latest IDNA standard from 2008 (!). IDNA 2003 is not compatible with German, Greek, Farsi and Sinhalese domains, http://unicode.org/faq/idn.html.

[abracadaber]

Yes, I misspelled, match_hostname() fails with ssl.CertificateError.

[abracadaber]

Christian, thanks a lot for your comment and for patch you provide. It becomes much clearer.
I'll be watching for bpo-17305.

[1st1]

Christian, what's the status on this one?

[tiran]

It's a big, complicated mess. I can't implement IDN support correctly because Python lacks UTS#46 and IDNA2008 support. I just found out that IDNA 2008 is not enough because it does not provide a case mapping. Lack of case mapping broke my fix for curl CVE-2016-8625.

At the moment IDN support is broken in a sane way: it just doesn't work and fails.

A partial fix will introduce security issues. http://unicode.org/reports/tr46/#Processing lists "www.sparkasse-gießen.de" as a critical example. It's the domain of a German savings and loan bank.

[kedare]

Hello Christian.

Is there any update about this issue ?
Do we have any alternative to avoid this problem ?

Thank you.

[njsmith]

If the SSL module followed the pattern of encoding all str to bytes at the edges while leaving bytes alone, and used exclusively bytes internally (and in this case by "bytes" I mean "bytes objects containing A-labels"), then it would at least fix this bug and also make it possible for library authors to implement their own IDNA handling. Right now if you pass in a pre-encoded byte-string, exactly what ssl.py needs to compare to the certificate, then ssl.py will convert it *back* to text :-(.

[tiran]

I have an idea for a different approach that can be applied to both ssl and socket module. Stay tuned to this station for a PEP broadcast!

[tialaramex]

I endorse njs' recommended fix here. Don't try to get clever, this is a security component, it should be the dumbest it can be possibly be while being correct, because if it's smarter it will probably be wrong.