test_ssl: test_wrong_cert_tls13() and test_pha_required_nocert() fail randomly on Windows

[pablogsal]

BPO	43921
Nosy	@tiran, @zooba, @pablogsal, @miss-islington
PRs	bpo-43921: ignore failing test_wrong_cert_tls13 on Windows (GH-25561) #25561 bpo-43921: also accept EOF in post-handshake auth test (GH-25574) #25574 bpo-43921: Fix test_ssl.test_pha_required_nocert() #26489 [3.10] bpo-43921: Fix test_ssl.test_pha_required_nocert() (GH-26489) #26494 bpo-43921: Debug test_ssl failures on Windows #26501 bpo-43921: Fix test_ssl.test_wrong_cert_tls13() on Windows #26502 [3.10] bpo-43921: Fix test_ssl.test_wrong_cert_tls13() on Windows (GH-26502) #26518 bpo-43921: Cleanup test_ssl.test_wrong_cert_tls13() #26520

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-04-23.10:56:18.501>
labels = ['expert-SSL', 'type-bug', 'tests', '3.10']
title = 'test_ssl: test_wrong_cert_tls13() and test_pha_required_nocert() fail randomly on Windows'
updated_at = <Date 2021-06-04.12:30:13.828>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2021-06-04.12:30:13.828>
actor = 'vstinner'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Tests', 'SSL']
creation = <Date 2021-04-23.10:56:18.501>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 43921
keywords = ['patch']
message_count = 28.0
messages = ['391677', '391717', '391725', '391775', '394317', '394318', '394320', '394340', '394343', '394346', '394397', '394398', '394399', '394400', '394407', '394408', '394916', '394924', '394944', '394955', '394967', '394968', '395011', '395043', '395044', '395045', '395072', '395078']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'steve.dower', 'pablogsal', 'miss-islington']
pr_nums = ['25561', '25574', '26489', '26494', '26501', '26502', '26518', '26520']
priority = None
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43921'
versions = ['Python 3.10']

[pablogsal]

https://buildbot.python.org/all/#/builders/405/builds/9

Extract from the logs:

The Buildbot has detected a new failure on builder AMD64 Windows8.1 Non-Debug 3.x while building python/cpython.
Full details are available at:
https://buildbot.python.org/all/#builders/405/builds/9

Buildbot URL: https://buildbot.python.org/all/

Worker for this Build: ware-win81-release

Build Reason: <unknown>
Blamelist: E-Paine <63801254+E-Paine@users.noreply.github.com>, Raymond Hettinger <rhettinger@users.noreply.github.com>, Simon Charette <charette.s@gmail.com>, Steve Dower <steve.dower@python.org>

BUILD FAILED: failed test (failure)

Summary of the results of the build (if available):
===================================================

== Tests result: FAILURE then FAILURE ==

395 tests OK.

10 slowest tests:

• test_multiprocessing_spawn: 2 min 43 sec
• test_concurrent_futures: 2 min 25 sec
• test_io: 1 min 43 sec
• test_largefile: 1 min 41 sec
• test_peg_generator: 1 min 39 sec
• test_mmap: 1 min 19 sec
• test_asyncio: 58.8 sec
• test_regrtest: 45.3 sec
• test_unparse: 44.1 sec
• test_tokenize: 41.1 sec

1 test failed:
test_ssl

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

1 re-run test:
test_ssl

Total duration: 9 min 15 sec

Captured traceback
==================

Traceback (most recent call last):
   File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_ssl.py", line 2333, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
   File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\ssl.py", line 518, in wrap_socket
    return self.sslsocket_class._create(
   File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\ssl.py", line 1070, in _create
    self.do_handshake()
   File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\ssl.py", line 1339, in do_handshake
    self._sslobj.do_handshake()
 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:969)


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_ssl.py", line 255, in wrapper
    return func(*args, **kw)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_ssl.py", line 3171, in test_wrong_cert_tls13
    self.fail("Use of invalid cert should have failed!")
AssertionError: Use of invalid cert should have failed!

Test report
===========

Failed tests:

• test_ssl

Failed subtests:

• test_wrong_cert_tls13 - test.test_ssl.ThreadedTests

Sincerely,
-The Buildbot

[tiran]

New changeset e047239 by Christian Heimes in branch 'master':
bpo-43921: ignore failing test_wrong_cert_tls13 on Windows (GH-25561)
e047239

[tiran]

Another TLS 1.3 client cert auth test is flaky, too.

https://buildbot.python.org/all/#/builders/577/builds/14/steps/4/logs/stdio

======================================================================
FAIL: test_pha_required_nocert (test.test_ssl.TestPostHandshakeAuth)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\pull_request.bolen-windows10\build\lib\test\test_ssl.py", line 4412, in test_pha_required_nocert
    s.recv(1024)
AssertionError: SSLError not raised

[tiran]

New changeset ce9a064 by Christian Heimes in branch 'master':
bpo-43921: also accept EOF in post-handshake auth test (GH-25574)
ce9a064

[pablogsal]

Unfortunately this is still failing sporadically. Last failure 5 days ago:

https://buildbot.python.org/all/#/builders/405/builds/153

[pablogsal]

The issue is marked as referred blocker, it won't block this beta release, but it will block the next, so we should decide how to proceed by then

[tiran]

I neither have a macOS nor a Windows machine to reproduce and debug the issue. Since I'm cannot reproduce the problem on Linux, I'm unable to debug and fix it.

[pablogsal]

Ok, but bear in mind that this will block the next beta and you are the expert in this area.

[pablogsal]

Also, this started to happen after the changes in PEP-644, and seems that reverting all changes related to the pep fixes the issue (at least I cannot reproduce it anymore after manually reverting the changes).