valgrind reports leaks for test_zipimport

[rkuska]

BPO	23769
Nosy	@vstinner, @serhiy-storchaka, @rkuska
Superseder	bpo-40077: Convert static types to heap types: use PyType_FromSpec()
Files	leak.py: not so minimal reproducer report: report from valgrind leak2.py leak3.py leak4.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-28.02:41:02.724>
created_at = <Date 2015-03-24.22:32:14.240>
labels = ['tests']
title = 'valgrind reports leaks for test_zipimport'
updated_at = <Date 2022-01-28.02:41:02.723>
user = 'https://github.com/rkuska'

bugs.python.org fields:

activity = <Date 2022-01-28.02:41:02.723>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-28.02:41:02.724>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2015-03-24.22:32:14.240>
creator = 'rkuska'
dependencies = []
files = ['38672', '38673', '38675', '38684', '50594']
hgrepos = []
issue_num = 23769
keywords = []
message_count = 6.0
messages = ['239193', '239194', '239196', '239238', '325731', '411963']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'rkuska']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '40077'
type = None
url = 'https://bugs.python.org/issue23769'
versions = ['Python 3.4']

[rkuska]

Leaks happen only when both testDoctestFile and testDoctestSuite are run.
Run with Python 3.4.2 and 3.4.1 with same result.

I have extracted those two tests into leak.py (attached).

$ valgrind --suppressions=/../cpython/Misc/valgrind-python.supp python3 leak.py
==17896== Memcheck, a memory error detector
==17896== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==17896== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==17896== Command: python3 leak.py
==17896==
==17896==
==17896== HEAP SUMMARY:
==17896== in use at exit: 1,599,328 bytes in 11,595 blocks
==17896== total heap usage: 283,757 allocs, 272,162 frees, 37,891,147 bytes allocated
==17896==
==17896== LEAK SUMMARY:
==17896== definitely lost: 30 bytes in 1 blocks
==17896== indirectly lost: 0 bytes in 0 blocks
==17896== possibly lost: 597,418 bytes in 2,319 blocks
==17896== still reachable: 1,001,880 bytes in 9,275 blocks
==17896== suppressed: 0 bytes in 0 blocks
==17896== Rerun with --leak-check=full to see details of leaked memory
==17896==
==17896== For counts of detected and suppressed errors, rerun with: -v
==17896== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Note that when I remove support.modules_cleanup(*modules_before) from leak.py valgrind reports no leaks (in original test_zipimport those are run in setUp and tearDown).

Output of
valgrind --suppressions=/home/rkuska/upstream/cpython/Misc/valgrind-python.supp --leak-check=yes -v python3 leak.py
attached as report.

[rkuska]

Summary for
valgrind python3 test_zipimport.py

==18608==
==18608== HEAP SUMMARY:
==18608== in use at exit: 1,596,390 bytes in 11,536 blocks
==18608== total heap usage: 343,849 allocs, 332,313 frees, 59,355,776 bytes allocated
==18608==
==18608== LEAK SUMMARY:
==18608== definitely lost: 90 bytes in 3 blocks
==18608== indirectly lost: 0 bytes in 0 blocks
==18608== possibly lost: 594,488 bytes in 2,258 blocks
==18608== still reachable: 1,001,812 bytes in 9,275 blocks
==18608== suppressed: 0 bytes in 0 blocks
==18608== Rerun with --leak-check=full to see details of leaked memory
==18608==
==18608== For counts of detected and suppressed errors, rerun with: -v
==18608== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[vstinner]

When calling gc.collect() after each test, I don't see any leak anymore.

doTest() has a small leak: it prepends a path to sys.path, but it never removes it.

Try attached leak2.py: it displays something like +254 kB. Uncomment the two following lines and the output will be close to +0 kB.

    #sys.path = old_path
#gc.collect()

[rkuska]

I tried leak2.py with valgrind, I've uncommented the lines you mentioned.

$ valgrind python3 leak2.py
==28421== Memcheck, a memory error detector
==28421== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==28421== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==28421== Command: python3 leak2.py
==28421==
==28421==
==28421== HEAP SUMMARY:
==28421== in use at exit: 1,599,354 bytes in 11,594 blocks
==28421== total heap usage: 284,971 allocs, 273,377 frees, 37,976,898 bytes allocated
==28421==
==28421== LEAK SUMMARY:
==28421== definitely lost: 0 bytes in 0 blocks
==28421== indirectly lost: 0 bytes in 0 blocks
==28421== possibly lost: 597,482 bytes in 2,319 blocks
==28421== still reachable: 1,001,872 bytes in 9,275 blocks
==28421== suppressed: 0 bytes in 0 blocks
==28421== Rerun with --leak-check=full to see details of leaked memory
==28421==
==28421== For counts of detected and suppressed errors, rerun with: -v
==28421== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

I've removed import readline from leak2.py, everything else left as in previous run (also ofc removed tracemalloc).

$ valgrind python3 leak3.py
==28515== Memcheck, a memory error detector
==28515== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==28515== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==28515== Command: python3 leak2.py
==28515==
==28515==
==28515== HEAP SUMMARY:
==28515== in use at exit: 1,599,384 bytes in 11,595 blocks
==28515== total heap usage: 285,648 allocs, 274,053 frees, 38,126,379 bytes allocated
==28515==
==28515== LEAK SUMMARY:
==28515== definitely lost: 30 bytes in 1 blocks
==28515== indirectly lost: 0 bytes in 0 blocks
==28515== possibly lost: 597,375 bytes in 2,317 blocks
==28515== still reachable: 1,001,979 bytes in 9,277 blocks
==28515== suppressed: 0 bytes in 0 blocks
==28515== Rerun with --leak-check=full to see details of leaked memory
==28515==
==28515== For counts of detected and suppressed errors, rerun with: -v
==28515== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Duh? Why does readline have this effect?

[serhiy-storchaka]

zipimport has been rewritten in pure Python (bpo-25711).

[vstinner]

valgrind reports leaks for test_zipimport

It's not strictly a memory leak: running the same code multiple time doesn't leak memory. It's just that Python allocates memory once and never releases it (doesn't releast it at exit).

---

leak4.py: Updated and simplified script.

If the code is run 10 times, it doesn't leak. With 1, 10 or 100 loops (LOOPS=100), Python still says the same at exit:

$ ./python -I -X showrefcount leak4.py 
[1574 refs, 664 blocks]

Using cannot_deallocate2.patch of bpo-46417, I can see that multiple static types are not cleared at Python exit:
---
Cannot deallocate type 'Exception': it still has subclasses

• error
  Cannot deallocate type 'BaseException': it still has subclasses
• Exception
  Cannot deallocate type 'dict': it still has subclasses
• collections.defaultdict
  Cannot deallocate type 'object': it still has subclasses
• dict
• BaseException
• itertools.accumulate
• itertools.combinations
• itertools.combinations_with_replacement
• itertools.cycle
• itertools.dropwhile
• itertools.takewhile
• itertools.islice
• itertools.starmap
• itertools.chain
• itertools.compress
• itertools.filterfalse
• itertools.count
• itertools.zip_longest
• itertools.pairwise
• itertools.permutations
• itertools.product
• itertools.repeat
• itertools.groupby
• itertools._grouper
• itertools._tee
• itertools._tee_dataobject
• collections.deque
• _collections._deque_iterator
• _collections._deque_reverse_iterator
• _collections._tuplegetter
• _struct.Struct
• _struct.unpack_iterator
  ---

The _collections, itertools and _struct extensions implement types as static types which are not cleared at Python exit. It explains why Python says that there are still 664 memory blocks allocated at Python exit (664 blocks).

I consider that this issue is a duplicate of bpo-40077.