SSL cert verify fail for "www.verisign.com"

[nagle]

BPO	23476
Nosy	@jcea, @pitrou, @giampaolo, @tiran, @ned-deily, @alex, @asmeurer, @sigmavirus24, @dstufft, @demianbrecht, @Lukasa
Files	ssltest.py: Test program to reproduce bug cacert.pem: Certificate file for testing. store.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-03-05.03:11:56.256>
created_at = <Date 2015-02-18.01:07:17.804>
labels = ['type-bug', 'library']
title = 'SSL cert verify fail for "www.verisign.com"'
updated_at = <Date 2015-03-20.12:11:44.970>
user = 'https://bugs.python.org/nagle'

bugs.python.org fields:

activity = <Date 2015-03-20.12:11:44.970>
actor = 'jcea'
assignee = 'none'
closed = True
closed_date = <Date 2015-03-05.03:11:56.256>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2015-02-18.01:07:17.804>
creator = 'nagle'
dependencies = []
files = ['38165', '38166', '38287']
hgrepos = []
issue_num = 23476
keywords = ['patch', 'needs review']
message_count = 29.0
messages = ['236158', '236159', '236160', '236168', '236318', '236321', '236327', '236506', '236509', '236510', '236511', '236512', '236513', '236518', '236520', '236548', '236976', '236977', '236978', '236982', '236983', '236984', '237238', '237253', '237254', '237273', '237288', '237291', '237292']
nosy_count = 15.0
nosy_names = ['jcea', 'janssen', 'nagle', 'pitrou', 'giampaolo.rodola', 'christian.heimes', 'ned.deily', 'alex', 'Aaron.Meurer', 'python-dev', 'icordasc', 'dstufft', 'demian.brecht', 'Lukasa', 'lac']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23476'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[nagle]

SSL certificate verification fails for "www.verisign.com" when using the cert list from Firefox. Other sites ("google.com", "python.org") verify fine.

This may be related to a known, and fixed, OpenSSL bug. See:

http://rt.openssl.org/Ticket/Display.html?id=2732&user=guest&pass=guest
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1014640

Some versions of OpenSSL are known to be broken for cases where there multiple valid certificate trees. This happens when one root cert is being phased out in favor of another, and cross-signing is involved.

Python ships with its own copy of OpenSSL on Windows. Tests
for "www.verisign.com"

Win7, x64:

Python 2.7.9 with OpenSSL 1.0.1j 15 Oct 2014. FAIL
Python 3.4.2 with OpenSSL 1.0.1i 6 Aug 2014. FAIL
openssl s_client -OpenSSL 1.0.1h 5 Jun 2014 FAIL

Ubuntu 14.04 LTS, x64, using distro's versions of Python:

Python 2.7.6 - test won't run, needs create_default_context
Python 3.4.0 with OpenSSL 1.0.1f 6 Jan 2014. FAIL
openssl s_client OpenSSL 1.0.1f 6 Jan 2014 PASS

That's with the same cert file in all cases. The OpenSSL version for Python programs comes from ssl.OPENSSL_VERSION.

The Linux situation has me puzzled. On Linux, Python is supposedly using the system version of OpenSSL. The versions match. Why do Python and the OpenSSL command line client disagree? Different options passed to OpenSSL by Python?

A simple test program and cert file are attached. Please try this in your environment.

[nagle]

Add cert file for testing. Source of this file is

http://curl.haxx.se/ca/cacert.pem

[nagle]

To try this with the OpenSSL command line client, use this shell command:

openssl s_client -connect www.verisign.com:443 -CAfile cacert.pem

This provides more detailed error messages than Python provides.

"verify error:num=20:unable to get local issuer certificate" is the OpenSSL error for "www.verisign.com". The corresponding Python error is "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)."

[lac]

I have this problem too.

Debian jessie/sid
Python 2.7.8 (default, Nov 18 2014, 14:57:17)
Python 3.4.2 (default, Nov 13 2014, 07:01:52)

[pitrou]

This may be related to a known, and fixed, OpenSSL bug.

Where do you see that the bug is fixed?

[lac]

In https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1014640
it says :

FIX:
Fixed in Ubuntu 14.04 apparently.
Openssl upstream, see http://rt.openssl.org/Ticket/Display.html?id=2732

But I think the person who wrote that launchpad note was mistaken, as
the rt.openssl.org ticket still is marked open when I looked at it.

[nagle]

The "fix" in Ubuntu was to the Ubuntu certificate store, which is a directory tree with one cert per file, with lots of symbolic links with names based on hashes to express dependencies. Python's SSL isn't using that. Python is taking in one big text file of SSL certs, with no link structure, and feeding it to OpenSSL.

This is an option at

 SSLContext.load_verify_locations(cafile=None, capath=None, cadata=None)

I've been testing with "cafile". "capath" is a path to a set of preprocessed certs laid out like the Ubuntu certificate store. It may be that the directory parameter works but the single-file parameter does not. It's possible to create such a directory from a single .pem file by splitting the big file into smaller files (the suggested tool is an "awk" script) and then running "c_rehash", which comes with OpenSSL. See "https://www.openssl.org/docs/apps/c_rehash.html"

So I tried a workaround, using Python 3.4.0 and Ubuntu 14.04 LTS. I broke up "cacert.pem" into one file per cert with the suggested "awk" script, and used "c_rehash" to build all the links, creating a directory suitable for "capath". It didn't help. Fails for "verisign.com", works for "python.org" and "google.com", just like the original single-file test. The "capath" version did exactly the same thing as the "cafile" version.

Python is definitely reading the cert file or directories; if I try an empty cert file or dir, everything fails, like it should.

Tried the same thing on Win7 x64. Same result. Tried the command line openssl tool using the cert directory. Same results as with the single file on both platforms.

So that's not it.

A fix to OpenSSL was proposed in 2012, but no action was taken:

http://rt.openssl.org/Ticket/Display.html?id=2732 at
"Wed Jun 13 17:15:04 2012 Arne Becker - Correspondence added".

Any ideas?

[pitrou]

Python's SSL isn't using that. Python is taking in one big text file
of SSL certs, with no link structure, and feeding it to OpenSSL.

Python's SSL is not "taking" anything:

>>> r = urlopen('https://www.verisign.com')
>>> r.read(10)
b' <!DOCTYPE'

It's only if you feed it that particular CA file that you get the issue:

>>> cafile = 'cacert.pem'
>>> r = urlopen('https://www.verisign.com', cafile=cafile)
[...]
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>

You can *also* feed it a CA directory by using the "CApath" argument (not "CAfile").

Now it remains to be seen why "openssl s_client" works with the file nevertheless.

[tiran]

John, neither Python nor OpenSSL are shipped with certificates.

Python uses certificates from operating system. We decided against our own certificate store because we wanted to avoid exactly this kind of trouble. If Python can't verify a certificate then you have to update the certificate storage of your OS.

On Linux and BSD Python, curl, wget and most other system tools use the OS's cert store. On Windows Python uses the same store as the IE, Chrome and other apps. Contrary to IE Python doesn't enforce cert store updates.

You can reproduce the problem with curl, too. The first call uses the OS' store, the second overwrite the default store.

$ curl https://www.verisign.com

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp curl https://www.verisign.com

[sigmavirus24]

So requests is running into this issue as well (see: https://github.com/kennethreitz/requests/issues/2455, https://github.com/kennethreitz/requests/issues/2456). With the specific code in Cory Benfield's comment (see: https://github.com/kennethreitz/requests/issues/2455#issuecomment-75773677) and the certificate file that requests 2.5.2 used (see: https://github.com/kennethreitz/requests/blob/d8be2473d1a586a3673d728d49e10fd4286e3b0e/requests/cacert.pem, raw: https://raw.githubusercontent.com/kennethreitz/requests/d8be2473d1a586a3673d728d49e10fd4286e3b0e/requests/cacert.pem) we can reproduce a similar problem on all versions of Python.

At the moment, we're investigating the possibility that it has to do with cross-signed certificates (see: http://openssl.6102.n7.nabble.com/Problems-with-cross-signed-certificates-and-Authority-Key-Info-td52280.html). We have a number of servers that we can reproduce this against and it is not reproducible using openssl s_client which means it is an issue with how Python has written its openssl compatibility layer.

[pitrou]

Ok, this is really a bug in the cert bundle provided by requests and Firefox.

With requests 2.5.1:

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp openssl s_client -CAfile requests/cacert.pem -connect verisign.com:443

=> ok

With requests 2.5.2:

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp openssl s_client -CAfile requests/cacert.pem -connect verisign.com:443

=> Verify return code: 20 (unable to get local issuer certificate)

[pitrou]

and it is not reproducible using openssl s_client

I have determined that s_client is buggy. It will always load the system certs *if and only if* you also pass it a valid custom CA cert (which is the reverse of what's expected).

This is where it happens (in apps/s_client.c):

if ((!SSL_CTX_load_verify_locations(ctx, CAfile, CApath)) ||
    (!SSL_CTX_set_default_verify_paths(ctx))) {
    /*
     * BIO_printf(bio_err,"error setting default verify locations\n");
     */
    ERR_print_errors(bio_err);
    /* goto end; */
}

This is why I forced SSL_CERT_* to empty locations in the examples above, so that only the custom CA bundle is used.