visit_decref(): add an assertion to check that the object is not freed

[vstinner]

BPO	38070
Nosy	@vstinner, @pablogsal, @miss-islington
PRs	bpo-38070: visit_decref() calls _PyObject_IsFreed() #15782 [3.8] bpo-38070: visit_decref() calls _PyObject_IsFreed() (GH-15782) #15793 bpo-38070: Enhance _PyObject_Dump() #16243 bpo-38070: _Py_DumpTraceback() writes <no Python frame> #16244 bpo-38070: Add _PyRuntimeState.preinitializing #16245 bpo-38070: Py_FatalError() logs runtime state #16246 [3.8] bpo-38070: Py_FatalError() logs runtime state #16258 bpo-38070: Enhance visit_decref() debug trace #16631 [3.7] bpo-38070: visit_decref() calls _PyObject_IsFreed() in debug mode #16816
Files	bpo-38037-bug.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-18.12:10:58.170>
created_at = <Date 2019-09-09.14:05:36.804>
labels = ['interpreter-core', 'type-feature', '3.8', '3.9']
title = 'visit_decref(): add an assertion to check that the object is not freed'
updated_at = <Date 2019-10-16.01:37:51.802>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-10-16.01:37:51.802>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-18.12:10:58.170>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-09-09.14:05:36.804>
creator = 'vstinner'
dependencies = []
files = ['48602']
hgrepos = []
issue_num = 38070
keywords = ['patch']
message_count = 15.0
messages = ['351470', '351473', '351504', '351537', '351645', '351686', '352665', '352666', '352667', '352675', '352676', '352720', '352721', '354156', '354681']
nosy_count = 3.0
nosy_names = ['vstinner', 'pablogsal', 'miss-islington']
pr_nums = ['15782', '15793', '16243', '16244', '16245', '16246', '16258', '16631', '16816']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38070'
versions = ['Python 3.8', 'Python 3.9']

[vstinner]

This issue is related to bpo-36389 "Add gc.enable_object_debugger(): detect corrupted Python objects in the GC.

I propose to call _PyObject_IsFreed() in visit_decref() in debug mode, to get a better error message if something goes wrong.

visit_decref() is commonly found in C traceback (ex: gdb backtrace) of bugs related to the garbage collector.

[vstinner]

visit_decref() is commonly found in C traceback (ex: gdb backtrace) of bugs related to the garbage collector.

Example with attached bpo-38037-bug.patch: you can apply this patch on the master branch to reintroduce bpo-38037 bug.

vstinner@apu$ git apply bpo-38037-bug.patch
vstinner@apu$ make

Without PR 15782:

vstinner@apu$ ./python -c pass
Segmentation fault (core dumped)

With PR 15782:

vstinner@apu$ ./python -c pass
Modules/gcmodule.c:379: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
<object: freed>
Fatal Python error: _PyObject_AssertFailed

Current thread 0x00007f171a280740 (most recent call first):
Aborted (core dumped)

The bug is catched earlier. And Python provides more information.

[vstinner]

New changeset d91d4de by Victor Stinner in branch 'master':
bpo-38070: visit_decref() calls _PyObject_IsFreed() (GH-15782)
d91d4de

[miss-islington]

New changeset 5731172 by Miss Islington (bot) in branch '3.8':
bpo-38070: visit_decref() calls _PyObject_IsFreed() (GH-15782)
5731172

[pablogsal]

Should this be closed now?

[vstinner]

Should this be closed now?

Please keep it open, I have a few more local changess to enhance Py_FatalError() :-) I would like to reuse the bpo number.

[vstinner]

New changeset 8fa3e17 by Victor Stinner in branch 'master':
bpo-38070: _Py_DumpTraceback() writes <no Python frame> (GH-16244)
8fa3e17

[vstinner]

New changeset b39afb7 by Victor Stinner in branch 'master':
bpo-38070: Enhance _PyObject_Dump() (GH-16243)
b39afb7

[vstinner]

New changeset d3b9041 by Victor Stinner in branch 'master':
bpo-38070: Add _PyRuntimeState.preinitializing (GH-16245)
d3b9041

[vstinner]

New changeset 1ce16fb by Victor Stinner in branch 'master':
bpo-38070: Py_FatalError() logs runtime state (GH-16246)
1ce16fb

[vstinner]

msg351473 example with the new assertion and enhanced debug functions:
---
Modules/gcmodule.c:379: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
<object at 0x7ff0038956d0 is freed>
Fatal Python error: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x1d1c9b0)

Current thread 0x00007ff010841740 (most recent call first):
<no Python frame>
Aborted (core dumped)
---

• Obviously, "Modules/gcmodule.c:379: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed" is the first most visible enhancement
• Add "Python runtime state: finalizing (tstate=0x1d1c9b0)": Python is finalizing (Py_Finalize)
• New "<no Python frame>" in the traceback
• "<Freed object>" became "<object at 0x7ff0038956d0 is freed>": add the object address

[vstinner]

New changeset 47bbab9 by Victor Stinner in branch '3.8':
[3.8] bpo-38070: Py_FatalError() logs runtime state (GH-16258)
47bbab9