test_ssl and test_httplib.HTTPSTest crash randomly with "Windows fatal exception: access violation" on Windows

[vstinner]

BPO	44252
Nosy	@db3l, @vstinner, @tiran, @pablogsal, @miss-islington, @erlend-aasland, @Fidget-Spinner
PRs	bpo-44252: Correctly implement gc support for SSLError objects #26439 [3.10] bpo-44252: Correctly implement gc support for SSLError objects (GH-26439) #26441

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2021-05-29.00:11:19.695>
created_at = <Date 2021-05-27.17:13:50.782>
labels = ['expert-SSL', 'tests', '3.11']
title = 'test_ssl and test_httplib.HTTPSTest crash randomly with "Windows fatal exception: access violation" on Windows'
updated_at = <Date 2021-05-29.00:11:19.694>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-29.00:11:19.694>
actor = 'pablogsal'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2021-05-29.00:11:19.695>
closer = 'pablogsal'
components = ['Tests', 'SSL']
creation = <Date 2021-05-27.17:13:50.782>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 44252
keywords = ['patch']
message_count = 28.0
messages = ['394573', '394581', '394599', '394600', '394605', '394606', '394608', '394614', '394615', '394617', '394619', '394621', '394624', '394654', '394655', '394661', '394685', '394687', '394688', '394691', '394693', '394694', '394695', '394696', '394698', '394699', '394703', '394704']
nosy_count = 7.0
nosy_names = ['db3l', 'vstinner', 'christian.heimes', 'pablogsal', 'miss-islington', 'erlendaasland', 'kj']
pr_nums = ['26439', '26441']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44252'
versions = ['Python 3.11']

[vstinner]

Christian Heimes reported a crash in bpo-42972:
https://bugs.python.org/issue42972#msg394520
---
#70587 is failing with an access violation on Windows. It's failing in one of the flaky tests. I wonder if the segfault is related to flaky tests somehow...

https://dev.azure.com/Python/cpython/_build/results?buildId=81570&view=logs&j=c83831cd-3752-5cc7-2f01-8276919eb334

test_pha_optional (test.test_ssl.TestPostHandshakeAuth) ... ok
test_pha_optional_nocert (test.test_ssl.TestPostHandshakeAuth) ... ok
test_pha_required (test.test_ssl.TestPostHandshakeAuth) ... ok
Windows fatal exception: access violation

Current thread 0x000009e0 (most recent call first):
File "D:\a\1\s\lib\linecache.py", line 63 in checkcache
File "D:\a\1\s\lib\traceback.py", line 375 in extract
File "D:\a\1\s\lib\traceback.py", line 494 in __init__
File "D:\a\1\s\lib\traceback.py", line 132 in format_exception
File "D:\a\1\s\lib\test\test_ssl.py", line 262 in handle_error
File "D:\a\1\s\lib\test\test_ssl.py", line 2530 in run
File "D:\a\1\s\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\1\s\lib\threading.py", line 963 in _bootstrap

Thread 0x000003c4 (most recent call first):
File "D:\a\1\s\lib\threading.py", line 1102 in _wait_for_tstate_lock
File "D:\a\1\s\lib\threading.py", line 1086 in join
File "D:\a\1\s\lib\test\test_ssl.py", line 2604 in run
File "D:\a\1\s\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\1\s\lib\threading.py", line 963 in _bootstrap

Thread 0x00001700 (most recent call first):
File "D:\a\1\s\lib\ssl.py", line 1131 in read
File "D:\a\1\s\lib\ssl.py", line 1256 in recv
File "D:\a\1\s\lib\test\test_ssl.py", line 4471 in test_pha_required_nocert
File "D:\a\1\s\lib\unittest\case.py", line 549 in _callTestMethod
File "D:\a\1\s\lib\unittest\case.py", line 592 in run
File "D:\a\1\s\lib\unittest\case.py", line 652 in __call__
File "D:\a\1\s\lib\unittest\suite.py", line 122 in run
File "D:\a\1\s\lib\unittest\suite.py", line 84 in __call__
File "D:\a\1\s\lib\unittest\suite.py", line 122 in run
File "D:\a\1\s\lib\unittest\suite.py", line 84 in __call__
File "D:\a\1\s\lib\unittest\runner.py", line 176 in run
File "D:\a\1\s\lib\test\support\init.py", line 959 in _run_suite
File "D:\a\1\s\lib\test\support\init.py", line 1082 in run_unittest
File "D:\a\1\s\lib\test\test_ssl.py", line 5007 in test_main
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 154 in _runtest
File "D:\a\1\s\lib\test\main.py", line 2 in <module>
File "D:\a\1\s\lib\runpy.py", line 86 in _run_code
File "D:\a\1\s\lib\runpy.py", line 196 in _run_module_as_main
##[error]Cmd.exe exited with code '-1073741819'.
---

Erlend E. Aasland added:
---
Hm, I'm unable to reproduce it w/addr sanitiser on macOS (FWIW).

$ ./python.exe -m test test_ssl -F -u all -m test_pha_required_nocert

Passing 1000 successful runs now. I'll see if I can get a Win dev env set up later.
---

I may be related to commit dcb8786 "bpo-42972: Fully implement GC protocol for ssl heap types (GH-26370)".

[erlend-aasland]

GH 26411, run id 2687367477 also failed with an access violation:
https://github.com/python/cpython/pull/26411/checks?check_run_id=2687367477

Not in test_pha_required_nocert, but test_local_bad_hostname.

May be related.

[vstinner]

Crash also seen on test_httplib:

(...)
test_attributes (test.test_httplib.HTTPSTest) ... ok
test_host_port (test.test_httplib.HTTPSTest) ... ok
Windows fatal exception: access violation

Current thread 0x000011ec (most recent call first):
File "D:\a\cpython\cpython\lib\socket.py", line 502 in close
File "D:\a\cpython\cpython\lib\ssl.py", line 1072 in _create
File "D:\a\cpython\cpython\lib\ssl.py", line 518 in wrap_socket
File "D:\a\cpython\cpython\lib\test\ssl_servers.py", line 37 in get_request
File "D:\a\cpython\cpython\lib\socketserver.py", line 311 in _handle_request_noblock
File "D:\a\cpython\cpython\lib\socketserver.py", line 237 in serve_forever
File "D:\a\cpython\cpython\lib\test\ssl_servers.py", line 142 in run
File "D:\a\cpython\cpython\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\cpython\cpython\lib\threading.py", line 963 in _bootstrap

Thread 0x000013ec (most recent call first):
File "D:\a\cpython\cpython\lib\socket.py", line 496 in _real_close
File "D:\a\cpython\cpython\lib\ssl.py", line 1330 in _real_close
File "D:\a\cpython\cpython\lib\socket.py", line 502 in close
File "D:\a\cpython\cpython\lib\ssl.py", line 1072 in _create
File "D:\a\cpython\cpython\lib\ssl.py", line 518 in wrap_socket
File "D:\a\cpython\cpython\lib\http\client.py", line 1448 in connect
File "D:\a\cpython\cpython\lib\http\client.py", line 969 in send
File "D:\a\cpython\cpython\lib\http\client.py", line 1031 in _send_output
File "D:\a\cpython\cpython\lib\http\client.py", line 1271 in endheaders
File "D:\a\cpython\cpython\lib\http\client.py", line 1322 in _send_request
File "D:\a\cpython\cpython\lib\http\client.py", line 1276 in request
File "D:\a\cpython\cpython\lib\test\test_httplib.py", line 1950 in test_local_bad_hostname
File "D:\a\cpython\cpython\lib\unittest\case.py", line 549 in _callTestMethod
File "D:\a\cpython\cpython\lib\unittest\case.py", line 592 in run
File "D:\a\cpython\cpython\lib\unittest\case.py", line 652 in __call__
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 122 in run
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 84 in __call__
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 122 in run
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 84 in __call__
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 122 in run
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 84 in __call__
File "D:\a\cpython\cpython\lib\unittest\runner.py", line 176 in run
File "D:\a\cpython\cpython\lib\test\support\init.py", line 959 in _run_suite
File "D:\a\cpython\cpython\lib\test\support\init.py", line 1082 in run_unittest
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 210 in _test_module
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 154 in _runtest
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 194 in runtest
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 321 in rerun_failed_tests
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 698 in _main
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 641 in main
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 719 in main
File "D:\a\cpython\cpython\lib\test\main.py", line 2 in <module>
File "D:\a\cpython\cpython\lib\runpy.py", line 86 in _run_code
File "D:\a\cpython\cpython\lib\runpy.py", line 196 in _run_module_as_main
test_local_bad_hostname (test.test_httplib.HTTPSTest) ...
Error: Process completed with exit code 1.

test.pythoninfo:

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 1.1.1k 25 Mar 2021
ssl.OPENSSL_VERSION_INFO: (1, 1, 1, 11, 15)
ssl.OP_ALL: -0x7fffffac
ssl.OP_NO_TLSv1_1: 0x10000000
ssl.SSLContext.maximum_version: MAXIMUM_SUPPORTED
ssl.SSLContext.minimum_version: TLSv1_2
ssl.SSLContext.options: OP_NO_COMPRESSION|OP_ENABLE_MIDDLEBOX_COMPAT|OP_CIPHER_SERVER_PREFERENCE|OP_NO_SSLv3|0x54
ssl.SSLContext.protocol: PROTOCOL_TLS_CLIENT
ssl.SSLContext.verify_mode: CERT_REQUIRED
ssl.default_https_context.maximum_version: MAXIMUM_SUPPORTED
ssl.default_https_context.minimum_version: TLSv1_2
ssl.default_https_context.options: OP_NO_COMPRESSION|OP_ENABLE_MIDDLEBOX_COMPAT|OP_CIPHER_SERVER_PREFERENCE|OP_NO_SSLv3|0x54
ssl.default_https_context.protocol: PROTOCOL_TLS_CLIENT
ssl.default_https_context.verify_mode: CERT_REQUIRED
ssl.stdlib_context.maximum_version: MAXIMUM_SUPPORTED
ssl.stdlib_context.minimum_version: TLSv1_2
ssl.stdlib_context.options: OP_NO_COMPRESSION|OP_ENABLE_MIDDLEBOX_COMPAT|OP_CIPHER_SERVER_PREFERENCE|OP_NO_SSLv3|0x54
ssl.stdlib_context.protocol: PROTOCOL_TLS_CLIENT
ssl.stdlib_context.verify_mode: CERT_NONE

[vstinner]

I forgot the link:
https://github.com/python/cpython/pull/26411/checks?check_run_id=2687367477

Oh right, Erlend already reported exactly the same crash :-)

[pablogsal]

Is this happening on the current 3.10 or just with a specific PR?