AMD64 Arch Linux Asan 3.x fails: command timed out: 1200 seconds without output

[vstinner]

BPO	42985
Nosy	@orsenthil, @vstinner, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-26.09:00:44.913>
created_at = <Date 2021-01-20.21:18:02.157>
labels = ['tests', '3.10']
title = 'AMD64 Arch Linux Asan 3.x fails: command timed out: 1200 seconds without output'
updated_at = <Date 2021-01-26.09:00:44.912>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-01-26.09:00:44.912>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-26.09:00:44.913>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2021-01-20.21:18:02.157>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42985
keywords = []
message_count = 10.0
messages = ['385373', '385416', '385417', '385419', '385428', '385459', '385469', '385487', '385697', '385698']
nosy_count = 3.0
nosy_names = ['orsenthil', 'vstinner', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42985'
versions = ['Python 3.10']

[vstinner]

The AMD64 Arch Linux Asan 3.x buildbot worker started to fail at build 262:
https://buildbot.python.org/all/#/builders/582/builds/262
-----------
...
0:39:09 load avg: 1.15 running: test_multiprocessing_forkserver (30.0 sec)
0:39:39 load avg: 1.64 running: test_multiprocessing_forkserver (1 min)
0:39:53 load avg: 1.50 [419/420] test_multiprocessing_forkserver passed (1 min 12 sec)
0:39:53 load avg: 1.50 [420/420] test_dynamicclassattribute passed

command timed out: 1200 seconds without output running (...)
process killed by signal 9
program finished with exit code -1
elapsedTime=3595.378040
-----------

[vstinner]

It seems like something changed on the buildbot, not in Python, since it also fails on 3.8 and 3.9.

AMD64 Arch Linux Asan 3.9:
https://buildbot.python.org/all/#builders/579/builds/105

AMD64 Arch Linux Asan 3.8:
https://buildbot.python.org/all/#builders/580/builds/86

IMO we should disable ASAN (handling of signals) at runtime when we trigger a crash on purpose (ex: faulthandler._sigsegv()).

[vstinner]

About SIGSEGV logs, one option is to use ASAN_OPTIONS="handle_segv=0".

[vstinner]

Documentation of ASAN_OPTIONS:
https://github.com/google/sanitizers/wiki/SanitizerCommonFlags
https://github.com/google/sanitizers/wiki/AddressSanitizerFlags

[orsenthil]

IMO we should disable ASAN (handling of signals) at runtime when we trigger a crash on purpose (ex: faulthandler._sigsegv()).

ASAN_OPTIONS="handle_segv=0".

Both sound reasonable. But not sure if they will resolve this crash tough.

[vstinner]

Both sound reasonable. But not sure if they will resolve this crash tough.

Many tests do crash *on purpose*. Example on test_concurrent_futures.py:

def _crash(delay=None):
    """Induces a segfault."""
    if delay:
        time.sleep(delay)
    import faulthandler
    faulthandler.disable()
    faulthandler._sigsegv()

Internally, faulthandler._sigsegv() disables crash reports.

There is also test.support.SuppressCrashReport context manager to disable crash reports. But I failed to find a way to disable ASAN signal handler at runtime.

It's possible to disable the ASAN signal handler at runtime using signal.signal(SIGSEGV, signal.SIG_DFT), but that should be done before Python installs its own signal handler for that, like before calling faulthandler.enable(). It would require to inject code in test_faulthandler to restore the default handler *before* calling faulthandler.enable(). Right now, test_faulthandler is skipped on the ASAN buildbots.

[pablogsal]

About SIGSEGV logs, one option is to use ASAN_OPTIONS="handle_segv=0".

Opened python/buildmaster-config#222

[vstinner]

For faulthandler.enable(), maybe we reset SIGSEGV signal handler to the default handler if __has_feature(address_sanitizer) is true:
https://clang.llvm.org/docs/AddressSanitizer.html#conditional-compilation-with-has-feature-address-sanitizer

But we cannot do that in faulthandler._sigsegv() since this function is used to test_faulthandler to check the signal handler installed by faulthandler previously.

Maybe we should add a function to test.support which resets the signal handler and then trigger a crash.

There are multiple functions which trigger crashes on purpose:

• _testcapi.crash_no_current_thread() => Py_FatalError()
• _testcapi.return_null_without_error() => Py_FatalError()
• _testcapi.return_result_wit_error() => Py_FatalError()
• _testcapi.negative_refcount() => Py_FatalError()
• _testcapi.pymem_buffer_overflow() => Py_FatalError()
• _testcapi.set_nomemory(0) is used to trigger a _PyErr_NormalizeException crash => Py_FatalError()
• etc.

Py_FatalError() calls abort() which raises SIGABRT signal, but ASAN doesn't catch this signal.

More generally, search for support.SuppressCrashReport usage in tests.

See also faulthandler_suppress_crash_report() C function.

[vstinner]

Pablo added ASAN_OPTIONS=handle_segv=0 env var to his buildbot worker:
python/buildmaster-config@3ae3e1b

[vstinner]

The AMD64 Arch Linux Asan 3.x buildbot worker started to fail at build 262:
https://buildbot.python.org/all/#/builders/582/builds/262

It no longer fails, so I close the issue:
https://buildbot.python.org/all/#/builders/582/builds/278