./configure --with-address-sanitizer adds -fsanitize flags to CFLAGS/LDFLAGS used by 3rd party C extensions

[vstinner]

./configure --with-address-sanitizer (ASAN) and similar options for MSAN and UBSAN adds compiler and linker options not only to build Python itself, but also to build 3rd party C extensions.

The flags should be added to "NODIST" variables instead.

Linked PRs

• gh-109575: Don't export -fsanitize compiler options #109576

[vstinner]

Example of the issue:

$ ./configure --with-pydebug --with-address-sanitizer CC=clang 
$ grep fsanitize Makefile
BASECFLAGS=	-fsanitize=address -fno-omit-frame-pointer  -fno-strict-overflow -Wsign-compare -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer
CONFIGURE_LDFLAGS=	-fsanitize=address 

I wrote PR #109576 to fix the issue.

[gpshead]

I don't know how much this matters.

I could also imagine people who wanted the flags exported for doing full builds of everything with selected sanitizers available?

Do we know of any distros that ship any sanitizer enabled builds as packages?

Or of people building a CPython configured this way as part of a larger CI test that then builds extensions to test?

[vstinner]

My experience with sanitizers is not great. Unless you have a whole system (!) built with sanitizers, you are quickly getting troubles with some libraries which are not aware of sanitizers.

OpenSSL and MSAN: openssl/openssl#17784

I strongly suspect it is a false positive caused by the fact that the openssl library you're linking against is not built with -fsanitize=memory.

libc stat() and MSAN: #91043 and llvm/llvm-project#54131

It's unclear to me if it's an issue in Python itself, Clang/GCC, or how the glibc was built. (...) Anyway, I close the issue.

@gpshead:

Do we know of any distros that ship any sanitizer enabled builds as packages?

I'm not aware of any Linux distribution shipping packages built with ASAN, MSAN, UBSAN or another sanitizer. It seems like it's more developers building their softwares with sanitizers, but try to ignore warnings/errors in other projects code.

---

Just building Python with ASAN quickly crash since ASAN thinks that Python is leaking memory. I'm not even talking about running any trivial code, just build Python.

git clean -fdx
./configure --with-address-sanitizer CC=clang
make SHELL="bash -x"

Output:

(...)
+ ./_bootstrap_python ./Programs/_freeze_module.py importlib.util ./Lib/importlib/util.py Python/frozen_modules/importlib.util.h
./_bootstrap_python ./Programs/_freeze_module.py runpy ./Lib/runpy.py Python/frozen_modules/runpy.h
+ ./_bootstrap_python ./Programs/_freeze_module.py importlib.machinery ./Lib/importlib/machinery.py Python/frozen_modules/importlib.machinery.h
+ ./_bootstrap_python ./Programs/_freeze_module.py runpy ./Lib/runpy.py Python/frozen_modules/runpy.h

=================================================================
==819456==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 20202 byte(s) in 412 object(s) allocated from:
    #0 0x4d1ca2 in __interceptor_malloc (/home/vstinner/python/main/_bootstrap_python+0x4d1ca2) (BuildId: 75c20068b0ade2526ba63d8eaeca3c90445e197a)
    #1 0x75de2d in PyUnicode_New /home/vstinner/python/main/Objects/unicodeobject.c:1244:24
    #2 0x762baa in _PyUnicode_FromUCS1 /home/vstinner/python/main/Objects/unicodeobject.c:2031:11
    #3 0x762baa in PyUnicode_FromKindAndData /home/vstinner/python/main/Objects/unicodeobject.c:2102:16
    #4 0x9b0f1d in r_object /home/vstinner/python/main/Python/marshal.c:1157:17
    #5 0x9b0b8a in r_object /home/vstinner/python/main/Python/marshal.c:1222:18
(...)

You have to explicitly tune ASAN to disable checking for memory leaks. Python Buildbot and GHA CI jobs use more configuration:

ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0"

[vstinner]

I could also imagine people who wanted the flags exported for doing full builds of everything with selected sanitizers available?

As an user, I would prefer to select which C extensions are built with the sanitizer.

[sunmy2019]

you are quickly getting troubles with some libraries which are not aware of sanitizers.

ASAN doesn't require all parts to be built with asan. MSAN does require.

[gpshead]

Indeed, in the MSAN use case we should continue propagating the flag as the only time someone is doing an MSAN build is if they're building a whole instrumented stack (loading non-MSAN instrumented code into an MSAN process will lead to MSAN failures without difficult to manage ignorelists setup beforehand).

Agreed that the others do not necessarily need to be propagated as UBSAN and ASAN are a lot closer to being stand alone instrumentation.

As a user, I would prefer to select which C extensions are built with the sanitizer.

Do we have non-core-dev users complaining about this?

Is sanitizer flag propagation to third party extension module building causing problems?

What's the motivation for changing it beyond feeling like it might be wrong?

[vstinner]

I closed my PR: #109576 (comment)