Support OpenSSL 1.1.1

[tiran]

BPO	32947
Nosy	@vstinner, @tiran, @benjaminp, @ned-deily, @alex, @dstufft, @stratakis, @hroncok
PRs	bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes #5663 [3.7] bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes (GH-5663) #5923 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-6976) #8760 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-5663) #8761 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 #11612 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 #11612 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 #11612 [2.7] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-8761) #11876

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2019-05-29.02:06:57.441>
created_at = <Date 2018-02-25.11:54:40.611>
labels = ['expert-SSL', '3.8', 'type-feature', '3.7']
title = 'Support OpenSSL 1.1.1'
updated_at = <Date 2019-05-29.02:06:57.440>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2019-05-29.02:06:57.440>
actor = 'ned.deily'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2019-05-29.02:06:57.441>
closer = 'ned.deily'
components = ['SSL']
creation = <Date 2018-02-25.11:54:40.611>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 32947
keywords = ['patch']
message_count = 21.0
messages = ['312804', '312897', '317529', '317575', '322490', '322795', '323526', '323901', '323903', '323909', '325793', '325795', '325926', '326719', '326720', '333990', '335623', '342616', '342617', '343853', '343854']
nosy_count = 10.0
nosy_names = ['janssen', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'alex', 'dstufft', 'kroeckx', 'cstratak', 'hroncok']
pr_nums = ['5663', '5923', '8760', '8761', '11612', '11612', '11612', '11876']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32947'
versions = ['Python 3.7', 'Python 3.8']

[tiran]

I'm using this ticket as an epos to track commits and required changes for OpenSSL 1.1.1 and TLS 1.3. Fixes need to be backported to 2.7 and 3.6 to 3.8. We might have to consider backports to 3.4 and 3.5, too.

If all goes to plan, OpenSSL 1.1.1 final is scheduled for 8th May 2018, https://www.openssl.org/policies/releasestrat.html . It will contain support for TLS 1.3. Python should either support TLS 1.3 by then or disable TLS 1.3 by default.

Fixes:

• bpo-20995 added TLS 1.3 cipher suite support
• bpo-29136 added OP_NO_TLSv1_3
• bpo-30622 fixes NPN guard for OpenSSL 1.1.1

Issues:

• A new option OP_ENABLE_MIDDLEBOX_COMPAT is enabled by default. We need to expose the flag to make test pass.
• TLS 1.3 has changed session handling. The current session code cannot handle TLS 1.3 session resumption.
• Threaded echo server and asynchat based tests are failing with TLS 1.3. I haven't analyzed the issue properly. It looks like the server thread dies when a handshake error occurs.

[tiran]

Ned, Benjamin

OpenSSL 1.1.1 is scheduled to be released just before 3.7.0rc1 will come out. I'd rather address as many issues now instead of adding last minute patches to the release candidate.

Once OpenSSL 1.1.1 is out and Python 3.7 supports it, we can discuss backports. 3.6 and 2.7 are going to need fixing, too.

[ned-deily]

Christian, I'm not sure how this issue now differs from bpo-33618 (and whether it can be closed as a duplicate) but, with the delay in OpenSSL 1.1.1 and as discussed over there, full 1.1.1 support will have to wait for 3.7.1 et al so I'm removing the "deferred blocker" status.

[tiran]

3.7 and 3.8 support OpenSSL 1.1.1-pre7-dev. For 3.6 and 2.7 I have to backport some test fixes and documentation.

I prefer to wait until both TLS 1.3 and OpenSSL 1.1.1 have been finalized. Once 1.1.1 is out, I'll fix the outstanding issues on master and then backport all fixes to 3.6 / 2.7.

[hroncok]

Note that Fedora 29 updated openssl from 1.1.0h to 1.1.1-0.pre8 and Python 3.7 tests are failing. Not 100% sure it's related, but full report at:

https://bugzilla.redhat.com/show_bug.cgi?id=1609291

[stratakis]

Yes test_poplib and test_ftplib on fedora rawhide when run against openssl 1.1.1 pre8. Haven't tried the pr7, but assuming that the tests were fine before here is the list of changes between pre7 and pre8:

openssl/openssl@OpenSSL_1_1_1-pre7...OpenSSL_1_1_1-pre8