test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)

[pablogsal]

BPO	35925
Nosy	@gpshead, @vstinner, @tiran, @benjaminp, @ned-deily, @ambv, @zooba, @JulienPalard, @stratakis, @pablogsal, @miss-islington
PRs	bpo-35925: Skip SSL tests that fail due to weak external certs. #13124 [3.7] bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124) #13139 [3.6] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) #13252 [2.7] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) #13253

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-29.02:12:18.543>
created_at = <Date 2019-02-07.01:45:26.293>
labels = ['3.7', '3.8', 'type-bug', 'tests']
title = 'test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)'
updated_at = <Date 2019-05-29.02:12:18.530>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2019-05-29.02:12:18.530>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-29.02:12:18.543>
closer = 'ned.deily'
components = ['Tests']
creation = <Date 2019-02-07.01:45:26.293>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 35925
keywords = ['patch']
message_count = 29.0
messages = ['334996', '335001', '335047', '335630', '335639', '335640', '335642', '335709', '335949', '335968', '336038', '336056', '336212', '336520', '336521', '340086', '340090', '341584', '341650', '341678', '341679', '341902', '341903', '341913', '342172', '342173', '342382', '342384', '343855']
nosy_count = 11.0
nosy_names = ['gregory.p.smith', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'lukasz.langa', 'steve.dower', 'mdk', 'cstratak', 'pablogsal', 'miss-islington']
pr_nums = ['13124', '13139', '13252', '13253']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35925'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[pablogsal]

Example failures

https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

---

Ran 105 tests in 2.477s

Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ... server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ... server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS server
joining HTTPS thread
ok

Multiple SSL failures, also old commits that previously succeeded fail now. This seems something in the buildbot itself. Gregory, do you know if something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l worker?

[gpshead]

FYI - the name of this bot is misleading. It is now Debian testing as of 18
hours ago instead of obsolete Ubuntu 14.04. I finally upgraded it.

Opens version says 1.1.1a.

--
blame half the typos on my phone.

On Wed, Feb 6, 2019, 5:45 PM Pablo Galindo Salgado <report@bugs.python.org
wrote:

New submission from Pablo Galindo Salgado <pablogsal@gmail.com>:

Example failures

https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------

> Traceback (most recent call last):
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py",
> line 1629, in test_networked_good_cert
>     h.request('GET', '/')
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1229, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1275, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1224, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1016, in _send_output
>     self.send(msg)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 956, in send
>     self.connect()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1391, in connect
>     self.sock = self._context.wrap_socket(self.sock,
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 405, in wrap_socket
>     return self.sslsocket_class._create(
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 853, in _create
>     self.do_handshake()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 1117, in do_handshake
>     self._sslobj.do_handshake()
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
> verify failed: EE certificate key too weak (_ssl.c:1055)
> 

---

Ran 105 tests in 2.477s

Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
(_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
(_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
(_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ... server
(('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ... server
(('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS
server
joining HTTPS thread
ok

Multiple SSL failures, also old commits that previously succeeded fail
now. This seems something in the buildbot itself. Gregory, do you know if
something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l
worker?

----------
components: Tests
messages: 334996
nosy: gregory.p.smith, pablogsal
priority: normal
severity: normal
status: open
title: test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and
ARMv7 Ubuntu 3.x buildbots
versions: Python 3.7, Python 3.8

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35925\>

---

[gpshead]

I had emailed Christian around the same time you filed this.

"""
The problem likely not related to your hardware. I guess it's caused by
tightened crypto polices. OpenSSL 1.1.1 has disabled some weak crypto.
Some platforms like Debian and RHEL require even larger key sizes or
have disable some algorithms. Does the test also fail with the env var
OPENSSL_CONF set to a non-existing path?
""" - christian.heimes

testing that theory... setting OPENSSL_CONF=/invalid-path does indeed "fix" (work around) the failures. Presumably by relaxing the default system constraints.

I could have that env var set for this buildbot and eliminate the failure. But do we _want_ to do that? Anyone who compiles CPython and tries to run the test suite on a modern system with such an OpenSSL configuration is going to see similar failures and likely come to us first asking about them.

It seems like we'd be better off adjusting our test suite to work around the constraints or disable them only for the duration of a test intentionally violating them?

[vstinner]

Does test_ssl pass on the master branch?

[gpshead]

Not on this debian buster bot. look back a couple comments, there is a workaround. it seems to be an OpenSSL configuration issue / test expectations issue.

I think we should ultimately get our test suite so that it passes in default OS distro OpenSSL configs.

That could mean any of an altered environment for some tests, or skipping some tests in such an environment, or changing some tests to fit within modern OpenSSL desired ciphersuite/protocol setting constrains constraints - or a mix of all three.

[gpshead]

release managers are free to defer this blocker. i'm just marking it as such for the purposes of making sure it is a conscious decision.

The problem is more likely with our test suite vs the environment than it is with CPython itself.

[gpshead]

FWIW I've just manually confirmed that running Python 2.7's test_ssl with OPENSSL_CONF=/invalid-path set passes on the debian buster buildbot host.