Make OpenSSL module compatible with OpenSSL 1.1.0

[tiran]

BPO	26470
Nosy	@pitrou, @giampaolo, @tiran, @alex, @zware, @dstufft, @matrixise, @aixtools, @yan12125
PRs	[3.4] bpo-26470: Port ssl and hashlib module to OpenSSL 1.1.0. #12211 [3.5] bpo-36576: Skip test_ssl and test_asyncio tests failing with OpenSSL 1.1.1 #12694
Files	patch-lang_python35-OpenSSL-1.1.0+LibreSSL: Revised patch for OpenSSL 1.1.0 support Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch: Patch for 2.7 Port-Python-s-SSL-module-to-OpenSSL-1.1.0-3.patch Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-4.patch Port-Python-s-SSL-module-to-OpenSSL-1.1.0-4.patch Port-Python-s-SSL-module-to-OpenSSL-1.1.0-5.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-08.14:19:46.658>
created_at = <Date 2016-03-02.12:14:53.739>
labels = ['type-security']
title = 'Make OpenSSL module compatible with OpenSSL 1.1.0'
updated_at = <Date 2019-04-05.08:27:39.831>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2019-04-05.08:27:39.831>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-08.14:19:46.658>
closer = 'christian.heimes'
components = []
creation = <Date 2016-03-02.12:14:53.739>
creator = 'christian.heimes'
dependencies = []
files = ['42480', '44231', '44232', '44296', '44297', '44360']
hgrepos = []
issue_num = 26470
keywords = ['patch']
message_count = 27.0
messages = ['261108', '261140', '261906', '263545', '263546', '263780', '263781', '269851', '272128', '272165', '273675', '273706', '273723', '273837', '273880', '273882', '273883', '273885', '274110', '274162', '274219', '274220', '274224', '274359', '274438', '274460', '274462']
nosy_count = 13.0
nosy_names = ['janssen', 'pitrou', 'giampaolo.rodola', 'christian.heimes', 'alex', 'python-dev', 'zach.ware', 'dstufft', 'matrixise', 'spil', 'Michael.Felt', 'yan12125', 'smpeepers']
pr_nums = ['12211', '12694']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue26470'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[tiran]

OpenSSL 1.1.0 is changing a lot of internals. Most structs are opaque, RC4 and SSLv2 are gone. I've a rough patch in my private repos. I'll submit the patch as soon as the code is compatible with OpenSSL 1.0.2, too.

master...tiran:feature/openssl110

[aixtools]

Since you are looking, maybe look at whether it is also libreSSL compatible?

[tiran]

Here is a first working patch. It requires 1.1.0-pre4. The failing ALPN test is caused by a regression in OpenSSL.

[spil]

Testing this patch on HardenedBSD/LibreSSL (base SSL libs replaced with LibreSSL)

[spil]

Checking version numbers to see if a feature is available is a bad practice. How can features ever be removed this way! Would be better to check for the feature itself (using autoconf).

The patch was mostly OK but any check for OPENSSL_VERSION_NUMBER for now also requires a negative check for LIBRESSL_VERSION_NUMBER as LibreSSL froze features at 1.0.1g.

Next to that, anything requiring compression (CRIME attack) should be guarded using and #infdef OPENSSL_NO_COMP.

This patch allowed me to build Python 3.5 with LibreSSL 2.3 (i.e. without SSLv3, Compression, RC4, SHA-0, etc)

[tiran]

The patch makes Python compatible with OpenSSL 1.1.0-pre6-dev from
git. The ssl and hashlib module are also compatible with OpenSSL 0.9.8zh,
1.0.1s, 1.0.2g as well as LibreSSL 2.3.3.

[tiran]

PS: The patch depends on openssl/openssl#979

[spil]

Can you please replace the HAVE_RAND_EGD bits with OPENSSL_NO_EGD as defined by both OpenSSL 1.1 and LibreSSL?

EGD default disabled https://github.com/openssl/openssl/blob/master/Configure#L363
EGD methods not available https://github.com/openssl/openssl/blob/master/include/openssl/rand.h#L61

[matrixise]

Hi Christian,

I have reviewed your patch, seems to be fine for me.

[tiran]

Stéphane, I have addressed your code review.

def __new__() no longer hard-codes protocol. We can change that in a later version of Python. OpenSSL has deprecated all SSL methods except of the generic TLS method. The TLS method was formerly known as SSLv23 method and does auto-negotiation of the latest supported method.

Lib/test/test_ssl.py:1183: LibreSSL does not support SSL_CA_PATH and SSL_CA_DIR env vars. I have changed the comment on the test.

Modules/_hashopenssl.c:127: _hashopenssl.c now does error checks on EVP digest copy. The copy operation can fail when an EVP ENGINE is involved.

HAS_FAST_PKCS5_PBKDF2_HMAC is defined in _hashopenssl.c. OpenSSL used to have a bad implementation of PKBDF2. I fixed it in 2013. The workaround is no longer required for OpenSSL >= 1.1.0. You can find more details in https://jbp.io/2015/08/11/pbkdf2-performance-matters/

[tiran]

OpenSSL 1.1.0 final was released a couple of hours ago. One test is failing because it uses 3DES. 1.1.0 has 3DES disabled by default.

[yan12125]

There are still quite a few references to PROTOCOL_SSLv23 in Doc/library/ssl.rst. Should they be updated as well?