SSLContext.load_verify_locations behavior on Windows and OSX

[tiran]

BPO	22449
Nosy	@ncoghlan, @pitrou, @giampaolo, @tiran, @ned-deily, @alex, @ambv, @hynek, @dstufft, @miss-islington, @LincolnPuzey
PRs	bpo-45457: Minor fix to documentation for SSLContext.load_default_certs. #28947 [3.10] bpo-45457: Minor fix to documentation for SSLContext.load_default_certs. (GH-28947) #29373 [3.9] bpo-45457: Minor fix to documentation for SSLContext.load_default_certs. (GH-28947) #29374
Files	win32_load_SSL_CERT_env.patch issue22449.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-10-03.21:34:23.903>
created_at = <Date 2014-09-20.10:35:39.225>
labels = ['extension-modules', 'type-bug', 'library']
title = 'SSLContext.load_verify_locations behavior on Windows and OSX'
updated_at = <Date 2021-11-02.21:53:50.655>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-11-02.21:53:50.655>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2014-10-03.21:34:23.903>
closer = 'python-dev'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2014-09-20.10:35:39.225>
creator = 'christian.heimes'
dependencies = []
files = ['36668', '36793']
hgrepos = []
issue_num = 22449
keywords = ['patch']
message_count = 10.0
messages = ['227150', '227203', '227209', '227211', '227215', '228375', '228389', '405553', '405555', '405557']
nosy_count = 13.0
nosy_names = ['ncoghlan', 'janssen', 'pitrou', 'giampaolo.rodola', 'christian.heimes', 'ned.deily', 'alex', 'lukasz.langa', 'python-dev', 'hynek', 'dstufft', 'miss-islington', 'LincolnPuzey']
pr_nums = ['28947', '29373', '29374']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22449'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[tiran]

The behavior of SSLContext.load_verify_locations is rather inconsistent across platforms:

On most POSIX platforms (Linux, BSD, non-Apple builds of OpenSSL) it loads certificates from predefined locations. The locations are defined during compile time and usually differ between vendors and platforms. My WiP "Improve TLS/SSL support" PEP lists all common locations and the packages that offer the certs. On these platforms SSL_CERT_DIR and SSL_CERT_FILE overwrite the location.

On Windows SSL_CERT_DIR and SSL_CERT_FILE are never taken into account by SSLContext.load_verify_locations because it doesn't call SSLContext.set_default_verify_paths(). The attached patch is a semi-fix for the problem. With the patch certs from SSL_CERT_DIR and SSL_CERT_FILE are only *added* to trusted root CA certs. The certs from Windows' cert store 'CA' and 'ROOT' are still loaded.

On OSX with Apple's custom build of OpenSSL SSL_CERT_DIR and SSL_CERT_FILE take effect. But there is a twist! In case a root CA cert is not found Apple's Trust Evaluation Agent (TEA) kicks in and looks up certs from Apple's keychain. It's almost the same situation as on Windows but more magical. In order to disable TEA one has to set the env var OPENSSL_X509_TEA_DISABLE=1 *before* the first cert is validated. After that the env var has no effect as the value is cached. Hynek has documted it in his blog: https://hynek.me/articles/apple-openssl-verification-surprises/

[alex]

Does this effect anything besides causing SSL_CERT_DIR and SSL_CERT_FILE to be respected?

[tiran]

'to respect' is the wrong term. </nitpicking>

The patch makes the code use the two env vars. Even with my patch the function doesn't respect SSL_CERT_DIR and SSL_CERT_FILE because Windows' cert store is still in use. On Linux and BSD the two env vars replace the default locations.

[pitrou]

Do we have tests for SSL_CERT_DIR and SSL_CERT_FILE being "respected"?
Otherwise, it might make sense to add one.

[tiran]

There is only a test case for ssl.get_default_verify_paths(), but not for SSLContext or SSLSocket.

[alex]

Attached patch also adds a test -- I'm not convinced this is the best way to test this, but I don't see any other way either.

[python-dev]

New changeset d9c52836aec8 by Benjamin Peterson in branch '2.7':
also use openssl envvars to find certs on windows (closes bpo-22449)
https://hg.python.org/cpython/rev/d9c52836aec8

New changeset e1f453e13f8d by Benjamin Peterson in branch '3.4':
also use openssl envvars to find certs on windows (closes bpo-22449)
https://hg.python.org/cpython/rev/e1f453e13f8d

New changeset f1113c568c60 by Benjamin Peterson in branch 'default':
merge 3.4 (bpo-22449)
https://hg.python.org/cpython/rev/f1113c568c60

[ambv]

New changeset 3551bf1 by LincolnPuzey in branch 'main':
bpo-45457: Minor fix to documentation for SSLContext.load_default_certs. (GH-28947)
3551bf1

[ambv]

New changeset 71f602b by Miss Islington (bot) in branch '3.10':
bpo-45457: Minor fix to documentation for SSLContext.load_default_certs. (GH-28947) (GH-29373)
71f602b

[ambv]

New changeset 4ab6e52 by Miss Islington (bot) in branch '3.9':
bpo-45457: Minor fix to documentation for SSLContext.load_default_certs. (GH-28947) (GH-29374)
4ab6e52