OS X system openssl deprecated - installer should build local libssl

[ned-deily]

BPO	17128
Nosy	@loewis, @birkenfeld, @ronaldoussoren, @orsenthil, @pitrou, @larryhastings, @tiran, @benjaminp, @ned-deily, @merwok, @sdrees, @dstufft
PRs	[2.7] Fix errant NEWS item: bpo-19019 -> bpo-17128 #6470

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2018-04-15.20:49:37.942>
created_at = <Date 2013-02-04.18:31:20.724>
labels = ['OS-mac', 'expert-SSL', 'build']
title = 'OS X system openssl deprecated - installer should build local libssl'
updated_at = <Date 2018-04-15.20:49:37.941>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2018-04-15.20:49:37.941>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2018-04-15.20:49:37.942>
closer = 'ned.deily'
components = ['Build', 'macOS', 'SSL']
creation = <Date 2013-02-04.18:31:20.724>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = ['204']
issue_num = 17128
keywords = []
message_count = 38.0
messages = ['181366', '181389', '181392', '181397', '181408', '181410', '181420', '181433', '181435', '181663', '181664', '181850', '181880', '183512', '189204', '189244', '192600', '192602', '192604', '192617', '192634', '192636', '192643', '193461', '209814', '212412', '212414', '212428', '214333', '232415', '232433', '232599', '269701', '276043', '309802', '312891', '315294', '315341']
nosy_count = 18.0
nosy_names = ['loewis', 'georg.brandl', 'ronaldoussoren', 'orsenthil', 'pitrou', 'larry', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'eric.araujo', 'python-dev', 'piotr.dobrogost', 'esc24', 'dilettant', 'dstufft', 'mlen', 'Kevin Ollivier', 'clupo']
pr_nums = ['6470']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue17128'
versions = ['Python 3.5']

[ned-deily]

Apple has deprecated use of openssl in OS X due to its unstable API between versions:

"If your app depends on OpenSSL, you should compile OpenSSL yourself and statically link a known version of OpenSSL into your app"

https://developer.apple.com/library/mac/#documentation/security/Conceptual/cryptoservices/GeneralPurposeCrypto/GeneralPurposeCrypto.html

Currently OS X ships with patched versions of libssl 0.9.7 and 0.9.8. The 32-bit python.org installer links with and dynamically loads 0.9.7 and the 64-/32-bit installer with 0.9.8.

build-installer.py should be enhanced to build and link with its own universal more up-to-date static libssl, as is done for several other OS X-supplied libraries. Since apparently the openssl upstream builds do not support OS X universal builds, build-installer.py will need to learn how to build each arch separately and lipo them together.

With the current discussion around security issues, are there features in openssl 1.x.x that warrant making this a release blocker for 2.7.4 and 3.2.4? I should be able to implement and test this over the next few days if so.

Setting to release blocker for release managers' decision.

[benjaminp]

As you are the MacOSX expert, I'm going to defer your judgement (and/or Ronald's). I don't think the release will be for several days at least, so you should have time to test.

[ronaldoussoren]

I'm not sure if it is worthwhile to switch right now. Apple does deprecate the use of OpenSSL, but there version does offer a feature that's not in the default tree: it verifies SSL certificates against the CA list in the system keychain.

This means that users that verify certificates (cert_reqs=CERT_REQUIRED in the ssl module) could see a regression when they don't specificy a custom CA list. Not having to maintain such a list manually is very convenient.

In the longer run I'd like to try if it is possible to implement the SSL module (and other extensions linking with openssl) using Apple's crypto APIs.

(Note that a clear disadvantage of the latter is that those APIs are "above" the unix layer and likely cause problems when you use fork(2) without exec(2)).

[ned-deily]

Yes, as we've discussed, using the Apple Crypto APIs would be nice longer-term assuming the compatibility issues can be managed: the set of available APIs appear to have been evolving over the past several OS X releases. But moving away from openssl seems out of scope for maintenance releases. The reason why I brought this up now is to try to determine if there are any newer ssl features that may be need to be batteries-included to deal with possible changes to Distutils and/or its users (pip, et al).

Managing the CA certificates certainly is another issue. We should investigate what Apple and third-party distributors of openssl on OS X do. It would be best to be able to use the system-supplied ones since they are actively managed by Apple and can be by the user.

Also, it would be good to know how the Python Windows distribution handles openssl and certificates. Martin? Brian?

[merwok]

Using the CA bundle from the OS sounds great, not only for Macs :)

[ned-deily]

Somewhat coincidentally, bpo-17129 addresses the topic of certificate management across multiple platforms.

[ronaldoussoren]

Replacing openssl by the supported crypto api's is something for 3.4 or even 3.5.

There is a way to keep the current functionality while still shipping a build of openssl: apply the patch that implements the feature to the upstream version when building it (the patch is available on opensource.apple.com, that's how I know that they do this in the first place).

Something that should be tested before this gets merged: what happens when a user installs pyOpenSSL with python 2.7.3 install (linked to system openssl) and then upgrades to 2.7.4 linked to a custom build of openssl without changing pyOpenSSL.

I wouldn't expect problems when looking at the documentation (there doesn't seem to be a way to transfer SSL state at the C level), and something similar can already happen: python is linked with a fairly old version of OpenSSL, and you get a later version when linking on a newer OSX release (hence a lot of users that download the binary installer and then install pyOpenSSL already have a version mismatch between the two extensions using openssl).

[tiran]

On Windows urllib.request.urlopen("http://www.google.com", cadefault=True) fails with "certificate verify failed". (tested with Python 3.3 64bit)

[ronaldoussoren]

It doesn't raise an exception on OSX (close to the tip of the default branch), both for http://www.google.com/ and https://www.google.com/

[ned-deily]

After spending some time on this, I'm downgrading this from release blocker status. First, no one has yet identified any immediate need for openssl 1.0.x features to support possible PyPI enhancements, which was my original concern. Second, since the openssl build system does not support OS X universal builds or SDKs and is not autoconf-based, it does not fit well into the current OS X installer build process. I have a working first cut of building the libs but there is more to do. Third, there is the open issue of how to make root certs available. Ronald, I'm probably missing something obvious here but I don't see which Apple patch you are referring to. Can you elaborate?

There is also the issue of government export restrictions that seems to always come up with crypto software. AFAICT, as of a couple of years ago, there is no longer any restriction on shipping openssl binaries with any encryption algorithm from the US to any other country. There are still a few well-known patent issue which seem easy to avoid. But I am not a lawyer.

Unless someone objects, I'm going to treat this as a new feature for now and, once ready, we can re-examine backporting.