Let OpenSSL verify hostname and IP address

[tiran]

BPO	31399
Nosy	@vstinner, @tiran, @jwilk, @ramikg
PRs	bpo-31399: Let OpenSSL verify hostname and IP address #3462 bpo-28414: ssl module idna test #5395 gh-89562: Remove hostflags from PySSLContext #28602

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2018-02-26.08:41:39.490>
created_at = <Date 2017-09-08.21:04:46.145>
labels = ['expert-SSL', '3.8', 'type-feature', '3.7']
title = 'Let OpenSSL verify hostname and IP address'
updated_at = <Date 2021-10-07.08:47:49.369>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-10-07.08:47:49.369>
actor = 'ramikg'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2018-02-26.08:41:39.490>
closer = 'christian.heimes'
components = ['SSL']
creation = <Date 2017-09-08.21:04:46.145>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 31399
keywords = ['patch']
message_count = 7.0
messages = ['301731', '310240', '310241', '310862', '311098', '311129', '312893']
nosy_count = 5.0
nosy_names = ['vstinner', 'christian.heimes', 'jwilk', 'Socob', 'ramikg']
pr_nums = ['3462', '5395', '28602']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31399'
versions = ['Python 3.7', 'Python 3.8']

[tiran]

Python should no longer attempt to verify hostname and ip addresses itself. OpenSSL 1.0.2 and newer is able to verify hostname and IP addresses itself. The new APIs are properly hooked into chain validation step. Hostname matching implements RFC 6125. CN matching and partial wildcards can be tuned with additional. The API is documented here: https://www.openssl.org/docs/man1.0.2/crypto/X509_VERIFY_PARAM_set1_host.html . X509_VERIFY_PARAM_set1_host is available since OpenSSL 1.0.2. LibreSSL 2.5.3+ implement the proper bits and pieces, too.

Why should we use OpenSSL rather than matching hostnames ourselves?

In the past, OpenSSL did not contain any code to perform host name
matching. Application were required to role their own implementation.
This caused code duplication and various security issues, because
it is far from trivial to cover all edge cases. Python had multiple
security issues just caused by incorrect or buggy hostname matching:

• Until Python 3.2 and 2.7.9, the ssl module was not capable of
  performing host name matching. ssl.match_hostname() was
  introduced in 3.2.0 and later back-ported to 2.7.9.
• Issue bpo-12000: Subject CN was ignored when a subject alternative
  name extension (SAN) was present without dNSName entries, thus
  violating RFC 2818.
• CVE-2013-2099: Multiple wildcard characters could be abused
  for Denial-of-Service attack in the re module.
• Issue bpo-17997: RFC 2818 was superseded by RFC 6125, which no longer
  allows multiple wildcard characters. Wildcards are only supported
  in the left-most label.
• Issue bpo-17997: ssl.match_hostname() did not implement partial
  wildcards of international domain names correctly.
• Issue bpo-18709: The ssl module used an inappropriate OpenSSL function
  to convert host names from ASN.1 to strings. A host name with an
  embedded NULL byte could be abused to trick validation.
• Issue bpo-17305: The ssl module does not handle IDNA 2008-encoded
  host names correctly. It converts from IDN A-label (ASCII
  compatible encoding) to IDN U-label (unicode) with Python's idna
  encoding, which is IDNA 2003-only.
• Issue bpo-30141: The host name is not verified when a SSLSocket is
  created with do_handshake_on_connect=False and the application
  causes an implicit handshake w/o calling do_handshake() explicitly.
• A SSLSocket performs host name matching after the handshake and
  during the handshake. In case of an invalid host name, a client
  is suppose to abort the connection with appropriate TLS alert.
  This causes two problem. For one the server is not informed about
  a problem with the certificate. Also an invalid host name does not
  prevent the client from sending a TLS client authentication
  cert to a malicious server. The cert typically contains personal
  information like username and department.

[tiran]

Bad news, LibreSSL is the worst. Even the latest release 2.6.4 does not implement https://www.openssl.org/docs/man1.0.2/crypto/X509_VERIFY_PARAM_set1_host.html or X509_VERIFY_PARAM_set_hostflags(). I don't get why it provides X509_check_host() but not X509_VERIFY_PARAM_set1_host().

[tiran]

libressl/portable#381

[tiran]

New changeset 61d478c by Christian Heimes in branch 'master':
bpo-31399: Let OpenSSL verify hostname and IP address (bpo-3462)
61d478c

[vstinner]

It seems like the commit 61d478c introduced a regression in test_ftplib: bpo-32706. Can you please take a look?

[tiran]

New changeset 66e5742 by Christian Heimes in branch 'master':
bpo-28414: ssl module idna test (bpo-5395)
66e5742

[tiran]

The feature has been implemented. I'll take care of the failing tests in bpo-32706.