SSL tests failed due to expired svn.python.org SSL certificate

[yan12125]

BPO	25940
Nosy	@malemburg, @birkenfeld, @larryhastings, @benjaminp, @alex, @skrah, @vadmium, @koobs, @jamadden, @yan12125, @ewdurbin
Files	svn-cert.patch: [obsolete] local-server.patch local-server.v2.patch: Change tests to use local server pythontest.patch: Use self-signed.pythontest.net pythontest-set-ca.patch: For pythontestdotnet repo use-pythontest.v2.patch: Use self-signed.pythontest.net local-server.v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vadmium'
closed_at = <Date 2016-03-27.04:22:18.240>
created_at = <Date 2015-12-24.20:47:46.458>
labels = ['tests']
title = 'SSL tests failed due to expired svn.python.org SSL certificate'
updated_at = <Date 2016-03-27.04:22:18.238>
user = 'https://github.com/yan12125'

bugs.python.org fields:

activity = <Date 2016-03-27.04:22:18.238>
actor = 'martin.panter'
assignee = 'martin.panter'
closed = True
closed_date = <Date 2016-03-27.04:22:18.240>
closer = 'martin.panter'
components = ['Tests']
creation = <Date 2015-12-24.20:47:46.458>
creator = 'yan12125'
dependencies = []
files = ['41488', '41512', '41515', '41532', '41589', '41590', '41630']
hgrepos = []
issue_num = 25940
keywords = ['patch']
message_count = 29.0
messages = ['256969', '256994', '256996', '256997', '256999', '257000', '257152', '257181', '257442', '257443', '257452', '257453', '257578', '257588', '257589', '257749', '257762', '258076', '258078', '258086', '258100', '258200', '258203', '258253', '258259', '258262', '258352', '262504', '262507']
nosy_count = 14.0
nosy_names = ['lemburg', 'georg.brandl', 'geertj', 'larry', 'benjamin.peterson', 'alex', 'skrah', 'python-dev', 'martin.panter', 'koobs', 'jmadden', 'yan12125', 'vincent-legoll', 'EWDurbin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue25940'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[yan12125]

The certificate of svn.python.org expires at Thu 24 Dec 2015 08:28:32 PM CST GMT, about 20 minutes ago. Please update its certificate or lots of tests in Lib\test\test_ssl.py fails with SSL: CERTIFICATE_VERIFY_FAILED.

[koobs]

I'm here from (duplicate) bpo-25950

[malemburg]

Regardless of the cert having expired, I think SSL tests either should use the main python.org site, e.g. https://www.python.org/ or the domain we specifically have for stdlib tests: http://pythontest.net/.

Benjamin: If you need a cert for pythontest.net, I can issue one next week.

Since the tests use CA Cert as root CA for the domain, I'm not sure whether getting a non-CA Cert for svn.python.org would be a good thing. If possible, I can issue one as well, but would have to know how svn.python.org is managed.

[alex]

I agree the tests shouldn't rely on a legacy domain like svn.python.org

In the mean time Ernest is working on getting a valid cert set up.

[alex]

There's good news and bad news, which do you want first?

Good news, great!

svn.python.org now has a certificate that's not expired, and it's even trusted by major trust stores.

Bad news?

The tests rely on the cert for svn.python.org specifically be a cacert one. IMO this is a bug in the tests and should be fixed.

[ewdurbin]

Currently svn.python.org is served from one of the last remaining "unmanaged" boxes in our infrastructure, dinsdale.python.org.

If a cacert signed certificate can be supplied, it can be installed. I see a tree of them in /etc/apache2/ssl/cacert/{2010,2011,2013}. Previously it seems that they were supplied by instantssl as i see certs from 2005 and 2007 there.

Tying ourselves to a single CA for these tests indefinitely doesn't seem to be the greatest idea though.

[python-dev]

New changeset 9d0c15425b59 by Stefan Krah in branch 'default':
Issue bpo-25940: Make buildbots usable again until a solution is found.
https://hg.python.org/cpython/rev/9d0c15425b59

[skrah]

Setting to blocker because I've disabled the offending tests.

[vadmium]

Assuming it is practical, would it be appropriate to convert the failing tests to connect to a local server running in a background Python thread? Perhaps using the existing ThreadedEchoServer. That way the tests wouldn’t require an internet connection, and the certificates used etc should be all contained within the Python source tree.

[vadmium]

In the meantime, this patch changes the tests to use the new DST root certificate. This seems to be the minimum to make the tests pass again, but someone else should probably review it because I’m not too experienced with SSL certificate stuff.

[malemburg]

On 04.01.2016 08:43, Martin Panter wrote:

In the meantime, this patch changes the tests to use the new DST root certificate. This seems to be the minimum to make the tests pass again, but someone else should probably review it because I’m not too experienced with SSL certificate stuff.

Thanks. I still think that we should use a more stable
resource for this other than svn.python.org, e.g. the
https://www.pythontest.net/ setup which was created specifically
for stdlib tests and which can more easily be adapted to
fit the needs of the tests than a public resource which is meant
for other purposes.

The local server idea may also work, but would have to run on
a non-privileged port.

[koobs]

Can't / Shouldn't these be mocked?

[vadmium]

Koobs: What thing are you suggesting to mock? The remote server perhaps, which is what I want to do too?

Here is a patch that implements my local SSL server idea by reusing ThreadedEchoServer.

There is one problem with local-server.patch, and I don’t know how to fix it properly. At the end of test_bio_handshake() and test_bio_read_write_data(), the unwrap() call keeps raising SSL_ERROR_SYSCALL and the test hangs in an infinite loop. I could work around this by making the server call unwrap() before shutting the socket down. But it would be better to fix ssl_io_loop() if anyone understands what is going wrong.