Python segfaults when configured with --with-pydebug --with-trace-refs

[pablogsal]

BPO	38400
Nosy	@vstinner, @elprans, @pablogsal, @miss-islington
PRs	bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed #16630 [3.8] bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630) #18072 [3.7] bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630) #18073

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-07.23:43:22.405>
created_at = <Date 2019-10-07.23:01:37.662>
labels = ['interpreter-core', '3.9']
title = 'Python segfaults when configured with --with-pydebug --with-trace-refs'
updated_at = <Date 2020-01-19.23:51:01.644>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2020-01-19.23:51:01.644>
actor = 'Elvis.Pranskevichus'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-07.23:43:22.405>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2019-10-07.23:01:37.662>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 38400
keywords = ['patch']
message_count = 9.0
messages = ['354151', '354152', '354153', '354171', '360279', '360280', '360281', '360282', '360283']
nosy_count = 4.0
nosy_names = ['vstinner', 'Elvis.Pranskevichus', 'pablogsal', 'miss-islington']
pr_nums = ['16630', '18072', '18073']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38400'
versions = ['Python 3.9']

[pablogsal]

configure --with-pydebug --with-trace-refs && make

Modules/gcmodule.c:378: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
<object at 0x5557629ca260 is freed>
Fatal Python error: _PyObject_AssertFailed
Python runtime state: preinitialized

Current thread 0x00007f6e2863e740 (most recent call first):
<no Python frame>
/bin/sh: line 5: 28078 Aborted (core dumped) ./python -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
make: * [Makefile:592: pybuilddir.txt] Error 1

[pablogsal]

This is a regression introduced by:

commit 6876257
Author: Victor Stinner <vstinner@python.org>
Date: Mon Oct 7 18:42:01 2019 +0200

bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612)

bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.

Modify the following functions to also work in release mode:

* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()

Other changes:

* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
  (equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
  with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
  attempt to render repr() at the end.

[pablogsal]

New changeset 36e33c3 by Pablo Galindo in branch 'master':
bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630)
36e33c3

[vstinner]

Notes for myself.

the visit_decref() has been updated to display repr() of the parent object, rather than display the "freed" object.

Before:
---
Modules/gcmodule.c:378: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
<object at 0x5557629ca260 is freed>
Fatal Python error: _PyObject_AssertFailed
Python runtime state: preinitialized

Current thread 0x00007f6e2863e740 (most recent call first):
<no Python frame>
---

After:
---
Modules/gcmodule.c:378: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
Enable tracemalloc to get the memory block allocation traceback

object address : 0x7fe2d1a52b40
object refcount : 1
object type : 0x740be0
object type name: dict
object repr : {'__repr__': <slot wrapper '__repr__' of 'weakref' objects>, '__hash__': <slot wrapper '__hash__' of 'weakref' objects>, '__call__': <slot wrapper '__call__' of 'weakref' objects>, '__lt__': <slot wrapper '__lt__' of 'weakref' objects>, '__le__': <slot wrapper '__le__' of 'weakref' objects>, '__eq__': <slot wrapper '__eq__' of 'weakref' objects>, '__ne__': <slot wrapper '__ne__' of 'weakref' objects>, '__gt__': <slot wrapper '__gt__' of 'weakref' objects>, '__ge__': <slot wrapper '__ge__' of 'weakref' objects>, '__init__': <slot wrapper '__init__' of 'weakref' objects>, '__new__': <built-in method __new__ of type object at 0x749500>, '__callback__': <member '__callback__' of 'weakref' objects>, '__doc__': None}

Fatal Python error: _PyObject_AssertFailed
Python runtime state: preinitialized

Current thread 0x00007fe2dea35740 (most recent call first):
<no Python frame>
---

The problem was that the None object was seen as "freed" because _ob_prev=_ob_next=NULL. This object doesn't seem to be part of the list of all objects, sys.getobjects(). I fixed Py_None, but then I got the same assertion error on a different (moduledef) object. I gave up, and Pablo and me fixed _PyObject_IsFreed() instead.

[elprans]

Since https://bugs.python.org/issue38070 was backported to 3.8, shouldn't this as well? 3.8 is currently broken otherwise.

[pablogsal]

Since https://bugs.python.org/issue38070 was backported to 3.8, shouldn't this as well? 3.8 is currently broken otherwise.

Thanks for the catch, Elvis!

[miss-islington]

New changeset a360070 by Miss Islington (bot) in branch '3.7':
bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630)
a360070

[miss-islington]

New changeset 4cdb758 by Miss Islington (bot) in branch '3.8':
bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630)
4cdb758

[elprans]

Thanks for the lightning-fast turnaround, Pablo!