Free lists are still used after being finalized (cleared)

[skrah]

BPO	40887
Nosy	@vstinner, @skrah
PRs	bpo-40887: Fix finalize_interp_clear() for free lists #20698 bpo-40887: Don't use finalized free lists #20700

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-06-08.00:15:40.459>
created_at = <Date 2020-06-06.08:23:56.632>
labels = ['interpreter-core', '3.10', 'performance']
title = 'Free lists are still used after being finalized (cleared)'
updated_at = <Date 2020-06-23.09:37:03.379>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2020-06-23.09:37:03.379>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-08.00:15:40.459>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-06-06.08:23:56.632>
creator = 'skrah'
dependencies = []
files = []
hgrepos = []
issue_num = 40887
keywords = ['patch']
message_count = 8.0
messages = ['370813', '370922', '370924', '370925', '370936', '370937', '370972', '372147']
nosy_count = 2.0
nosy_names = ['vstinner', 'skrah']
pr_nums = ['20698', '20700']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue40887'
versions = ['Python 3.10']

[skrah]

I'm opening a separate issue to prevent bpo-40521 from getting too big.

Lists and tuples sometimes leak (starting 69ac6e5 and later):

==1445== 56 bytes in 1 blocks are definitely lost in loss record 1,542 of 4,898
==1445== at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
==1445== by 0x550487: _PyObject_GC_Alloc (gcmodule.c:2233)
==1445== by 0x550487: _PyObject_GC_Malloc (gcmodule.c:2260)
==1445== by 0x550487: _PyObject_GC_New (gcmodule.c:2272)
==1445== by 0x44CB04: PyList_New (listobject.c:144)
==1445== by 0x4E3DE1: init_filters (_warnings.c:88)
==1445== by 0x4E3DE1: warnings_init_state (_warnings.c:120)
==1445== by 0x4E3DE1: _PyWarnings_InitState (_warnings.c:1372)
==1445== by 0x521720: pycore_init_import_warnings (pylifecycle.c:687)
==1445== by 0x521720: pycore_interp_init (pylifecycle.c:735)
==1445== by 0x5246A0: pyinit_config (pylifecycle.c:763)
==1445== by 0x5246A0: pyinit_core (pylifecycle.c:924)
==1445== by 0x5246A0: Py_InitializeFromConfig (pylifecycle.c:1134)
==1445== by 0x4285DC: pymain_init (main.c:66)
==1445== by 0x4296A1: pymain_main (main.c:653)
==1445== by 0x4296A1: Py_BytesMain (main.c:686)
==1445== by 0x578882F: (below main) (libc-start.c:291)

==1445== 64 bytes in 1 blocks are definitely lost in loss record 2,259 of 4,898
==1445== at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
==1445== by 0x550611: _PyObject_GC_Alloc (gcmodule.c:2233)
==1445== by 0x550611: _PyObject_GC_Malloc (gcmodule.c:2260)
==1445== by 0x550611: _PyObject_GC_NewVar (gcmodule.c:2289)
==1445== by 0x48452C: tuple_alloc (tupleobject.c:76)
==1445== by 0x48452C: _PyTuple_FromArray (tupleobject.c:413)
==1445== by 0x435EE0: _PyObject_MakeTpCall (call.c:165)
==1445== by 0x436947: _PyObject_FastCallDictTstate (call.c:113)
==1445== by 0x436947: PyObject_VectorcallDict (call.c:142)
==1445== by 0x61DFC5: builtin___build_class__ (bltinmodule.c:232)
==1445== by 0x5E8A39: cfunction_vectorcall_FASTCALL_KEYWORDS (methodobject.c:440)
==1445== by 0x41F4D5: _PyObject_VectorcallTstate (abstract.h:114)
==1445== by 0x41F4D5: PyObject_Vectorcall (abstract.h:123)
==1445== by 0x41F4D5: call_function (ceval.c:5111)
==1445== by 0x42220E: _PyEval_EvalFrameDefault (ceval.c:3542)
==1445== by 0x4E6882: _PyEval_EvalFrame (pycore_ceval.h:40)
==1445== by 0x4E6882: _PyEval_EvalCode (ceval.c:4366)
==1445== by 0x4E6A65: _PyEval_EvalCodeWithName (ceval.c:4398)

[vstinner]

Lists and tuples sometimes leak (starting 69ac6e5 and later):

It's not a regression. It's just that bpo-40521 helped Valgrind to detect such bug :-)

I wrote bpo-20698 to fix this issue.

I'm also working on a more generic approach to ensure that we don't add new objects to free lists after free lists are cleared.

[vstinner]

I'm curious about this bug, so I looked at code changes.

Before Python 3.6, warnings.c "garbage" is declared as a static variable. It is initialized once but never cleared.

In Python 3.7, the variable moved into _PyRuntimeState.gc.

In Python 3.8, _PyGC_Fini() started to clear the "garbage" list. This function is called by Py_FinalizeEx() *after* PyList_Fini().

[vstinner]

New changeset 7907f8c by Victor Stinner in branch 'master':
bpo-40887: Fix finalize_interp_clear() for free lists (GH-20698)
7907f8c

[vstinner]

New changeset bcb1983 by Victor Stinner in branch 'master':
bpo-40887: Don't use finalized free lists (GH-20700)
bcb1983

[vstinner]

Thanks Stefan for the bug report. It's now fixed and I made sure that such bug cannot strike back.