Failure to build with address sanitizer

[btharper]

BPO	36356
Nosy	@vstinner, @ned-deily, @matrixise, @miss-islington, @btharper
PRs	bpo-36356: Fix leak of locale string #12425 bpo-36356: Release Unicode interned strings on Valgrind #12431 bpo-36333, bpo-36356: Fix _PyEval_FiniThreads() #12432 bpo-36356: pymain_free() calls _PyRuntime_Finalize() #12435 [3.7] bpo-36356: pymain_free() calls _PyRuntime_Finalize() #12436 bpo-36356: Fix leak when updating search path in config #12444 bpo-36356: Destroy the GIL at exit #12453 bpo-36356: Fix _PyCoreConfig_Read() #12454 bpo-36356: pymain_exit_error() only call pymain_free() for exit #12968 Revert "bpo-36356: Destroy the GIL at exit (GH-12453)" #13006 bpo-36356: Fix memory leak in _asynciomodule.c during "setup.py build_ext" #16598 [3.8] bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598) #16622

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-08.13:03:16.055>
created_at = <Date 2019-03-19.03:47:57.667>
labels = ['build', '3.8']
title = 'Failure to build with address sanitizer'
updated_at = <Date 2019-10-08.13:03:16.054>
user = 'https://github.com/btharper'

bugs.python.org fields:

activity = <Date 2019-10-08.13:03:16.054>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-08.13:03:16.055>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-03-19.03:47:57.667>
creator = 'btharper'
dependencies = []
files = []
hgrepos = []
issue_num = 36356
keywords = ['patch']
message_count = 28.0
messages = ['338315', '338326', '338329', '338338', '338341', '338352', '338353', '338357', '338358', '338364', '338378', '338431', '338774', '338776', '338907', '340907', '340912', '341060', '341061', '341064', '341065', '341066', '354107', '354116', '354119', '354123', '354200', '354201']
nosy_count = 5.0
nosy_names = ['vstinner', 'ned.deily', 'matrixise', 'miss-islington', 'btharper']
pr_nums = ['12425', '12431', '12432', '12435', '12436', '12444', '12453', '12454', '12968', '13006', '16598', '16622']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue36356'
versions = ['Python 3.8']

[btharper]

Trying to run make after './configure --with-address-sanitizer --with-pydebug' fails with leak of locale string

[matrixise]

just for me, btharper, which system do you use? because I have a fedora 29 and when I try to compile with these flags, getaddrinfo is not found.

[matrixise]

ok, found

I have to disable ipv6 and install libasan

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi

=================================================================
==5408==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 34 byte(s) in 1 object(s) allocated from:
#0 0x7fb085e03c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
#1 0x4d99a2 in _PyMem_RawMalloc Objects/obmalloc.c:99
#2 0x4d9b4a in _PyMem_DebugRawAlloc Objects/obmalloc.c:2020
#3 0x4d9ba5 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2049
#4 0x4dc354 in PyMem_RawMalloc Objects/obmalloc.c:527
#5 0x4dc510 in _PyMem_RawStrdup Objects/obmalloc.c:613
#6 0x652f5f in get_ctype_locale Python/preconfig.c:483
#7 0x6560a6 in _PyPreConfig_Read Python/preconfig.c:506
#8 0x616cde in _PyCoreConfig_ReadPreConfig Python/coreconfig.c:1342
#9 0x620564 in _PyCoreConfig_Read Python/coreconfig.c:1378
#10 0x65ef06 in pyinit_coreconfig Python/pylifecycle.c:740
#11 0x65fa05 in _Py_InitializeCore Python/pylifecycle.c:786
#12 0x42abba in pymain_init Modules/main.c:370
#13 0x42addc in pymain_main Modules/main.c:889
#14 0x42b2bb in _Py_UnixMain Modules/main.c:940
#15 0x4258ee in main Programs/python.c:16
#16 0x7fb085983412 in __libc_start_main ../csu/libc-start.c:308

Direct leak of 34 byte(s) in 1 object(s) allocated from:
#0 0x7fb085e03c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
#1 0x4d99a2 in _PyMem_RawMalloc Objects/obmalloc.c:99
#2 0x4d9b4a in _PyMem_DebugRawAlloc Objects/obmalloc.c:2020
#3 0x4d9ba5 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2049
#4 0x4dc354 in PyMem_RawMalloc Objects/obmalloc.c:527
#5 0x4dc510 in _PyMem_RawStrdup Objects/obmalloc.c:613
#6 0x652f5f in get_ctype_locale Python/preconfig.c:483
#7 0x6560a6 in _PyPreConfig_Read Python/preconfig.c:506
#8 0x6585f8 in pyinit_preconfig Python/pylifecycle.c:723
#9 0x65f8b0 in _Py_InitializeCore Python/pylifecycle.c:781
#10 0x42abba in pymain_init Modules/main.c:370
#11 0x42addc in pymain_main Modules/main.c:889
#12 0x42b2bb in _Py_UnixMain Modules/main.c:940
#13 0x4258ee in main Programs/python.c:16
#14 0x7fb085983412 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: 68 byte(s) leaked in 2 allocation(s).
generate-posix-vars failed
make: *** [Makefile:586: pybuilddir.txt] Error 1

[matrixise]

your PR seems to be fine.

but I continue to get two refleaks with valgrind but don't worry, @vstinner is working on these refleaks.

==5440== 64 bytes in 1 blocks are possibly lost in loss record 1 of 2
==5440== at 0x483880B: malloc (vg_replace_malloc.c:309)
==5440== by 0x46BFAB: _PyMem_RawMalloc (obmalloc.c:99)
==5440== by 0x46BEA3: _PyMem_DebugRawAlloc (obmalloc.c:2020)
==5440== by 0x46BF33: _PyMem_DebugRawMalloc (obmalloc.c:2049)
==5440== by 0x46D304: PyMem_RawMalloc (obmalloc.c:527)
==5440== by 0x52C1EF: PyThread_allocate_lock (thread_pthread.h:330)
==5440== by 0x5154F0: _PyRuntimeState_Init_impl (pystate.c:52)
==5440== by 0x515A76: _PyRuntimeState_Init (pystate.c:77)
==5440== by 0x514114: _PyRuntime_Initialize (pylifecycle.c:88)
==5440== by 0x4236FE: pymain_init (main.c:339)
==5440== by 0x423980: pymain_main (main.c:889)
==5440== by 0x423A6A: _Py_UnixMain (main.c:940)
==5440==
==5440== 64 bytes in 1 blocks are possibly lost in loss record 2 of 2
==5440== at 0x483880B: malloc (vg_replace_malloc.c:309)
==5440== by 0x46BFAB: _PyMem_RawMalloc (obmalloc.c:99)
==5440== by 0x46BEA3: _PyMem_DebugRawAlloc (obmalloc.c:2020)
==5440== by 0x46BF33: _PyMem_DebugRawMalloc (obmalloc.c:2049)
==5440== by 0x46D304: PyMem_RawMalloc (obmalloc.c:527)
==5440== by 0x52C1EF: PyThread_allocate_lock (thread_pthread.h:330)
==5440== by 0x515506: _PyRuntimeState_Init_impl (pystate.c:58)
==5440== by 0x515A76: _PyRuntimeState_Init (pystate.c:77)
==5440== by 0x514114: _PyRuntime_Initialize (pylifecycle.c:88)
==5440== by 0x4236FE: pymain_init (main.c:339)
==5440== by 0x423980: pymain_main (main.c:889)
==5440== by 0x423A6A: _Py_UnixMain (main.c:940)

[vstinner]

New changeset e130a07 by Victor Stinner (btharper) in branch 'master':
bpo-36356: Fix memory leak in _PyPreConfig_Read() (GH-12425)
e130a07

[vstinner]

New changeset a712679 by Victor Stinner in branch 'master':
bpo-36333, bpo-36356: Fix _PyEval_FiniThreads() (GH-12432)
a712679

[vstinner]

New changeset fecc4f2 by Victor Stinner in branch 'master':
bpo-36356: Release Unicode interned strings on Valgrind (bpo-12431)
fecc4f2

[vstinner]

New changeset f5f336a by Victor Stinner in branch 'master':
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12435)
f5f336a

[vstinner]

New changeset 935250d by Victor Stinner in branch '3.7':
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12436)
935250d

[vstinner]

At least, "./python -V" no longer leaks at commit dcf6171.

$ ./configure --with-valgrind
$ make
$ valgrind ./python -V
==9553== Memcheck, a memory error detector
==9553== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==9553== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==9553== Command: ./python -V
==9553== 
Python 3.8.0a2+
==9553== 
==9553== HEAP SUMMARY:
==9553==     in use at exit: 0 bytes in 0 blocks
==9553==   total heap usage: 33 allocs, 33 frees, 5,933 bytes allocated
==9553== 
==9553== All heap blocks were freed -- no leaks are possible
==9553== 
==9553== For counts of detected and suppressed errors, rerun with: -v
==9553== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[btharper]

I'm on Ubuntu 18.10/amd64 compiling with Ubuntu's GCC 8.2.0, I know there's some libraries that are missing dependencies (including bz2 and sqlite) so I may have missed the ipv6 dependencies as well.

My eventual goal was to be able to build a pgo optimized build of cpython with the address sanitizer turned on. Currently on my build of f5f336a I see 12 failing tests and a heap use after free as part of the ctypes test suite.

The ctypes use after free has already been submitted as bpo-36253 since it was the only thing asan caught during tests with the leak sanitizer turned off.

[vstinner]

New changeset 4a1468e by Victor Stinner in branch 'master':
bpo-36356: Fix _PyCoreConfig_Read() (GH-12454)
4a1468e