gh-140815: Fix faulthandler for invalid/freed frame #140921

vstinner · 2025-11-02T18:15:12Z

faulthandler now detects if a frame or a code object is invalid or freed.

Add helper functions:

_PyCode_SafeAddr2Line()
_PyFrame_SafeGetCode()
_PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now also considers the pointer 0x1 as freed.

Issue: SEGV in faulthandler.dump_traceback_later #140815

faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetBytecode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now also considers the pointer 0x1 as freed.

Include/internal/pycore_interpframe.h

Python/traceback.c

sergey-miryanov · 2025-11-02T18:58:40Z

I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution.

sergey-miryanov · 2025-11-02T19:07:12Z

Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT?

Include/internal/pycore_interpframe.h

vstinner · 2025-11-02T20:10:28Z

Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT?

I don't think that it's worth it.

Another faulthandler feature is faulthandler.enable() and faulthandler.register(): register signal handlers. A Unix signal can occur anytime, so we cannot acquire the GIL or any other lock.

faulthandler is a debugger, it's not designed to be 100% accurate, but to "just work" in any case (GIL or not, in a signal handler or not, etc.) without taking any lock.

vstinner · 2025-11-02T20:17:43Z

@sergey-miryanov @picnixz: I pushed a change to address your reviews.

Include/internal/pycore_code.h

efimov-mikhail · 2025-11-02T20:31:50Z

LGTM, but I have very little expertise here, so I'll not push approve button :-).

vstinner · 2025-11-02T20:32:28Z

I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution.

faulthandler is implemented with multiple heuristics and best effort. It's ok if it does crash, since it should only be triggered when something already goes wrong.

Only faulthandler.dump_traceback() should be safe since it's called with an attached thread safe.

sergey-miryanov

From point of my little experience, it looks good to me.

sergey-miryanov · 2025-11-02T20:44:21Z

Thanks!

Python/traceback.c

ZeroIntensity · 2025-11-02T21:23:41Z

Include/internal/pycore_interpframe.h

+    if (_PyMem_IsPtrFreed(ptr)) {
+        return NULL;
+    }
+    PyObject *executable = PyStackRef_AsPyObjectBorrow(f->f_executable);
+    if (_PyObject_IsFreed(executable)) {


Does this patch work on release builds? I thought _PyMem_IsPtrFreed and _PyObject_IsFreed only worked if debug allocators are enabled.

Oh. I didn't try on a release build! _PyMem_IsPtrFreed() and _PyObject_IsFreed() work better with debug allocators, you're correct. But they detect freed (reused) memory in some cases in a release build as well. They are just less reliable in release mode.

I pushed more changes to make the heuristics stricter.

vstinner · 2025-11-03T12:25:03Z

@picnixz @ZeroIntensity: Do these changes look reasonable to you? Even if they are not perfect, they allow dumping the traceback in more cases.

I wrote a script to run the reproducer in a loop (see below). I ran the stress test 3 times.

Debug build:
- Without these changes: crash after 1 run, crash after 6 runs, crash after 1 run
- With these changes: crash after 3970 runs, <stopped after 10,000 success without crash>, <stopped after 10,000 success without crash>.
Release build:
- Without these changes: crash after 6 runs, crash after 59 runs, crash after 36 runs
- With these changes: crash after 550 runs, crash after 3750 runs, crash after 7845 runs.

Stress test:

import subprocess, sys
cmd=[sys.executable, 'reproducer.py']
success = 0
while True:
    proc = subprocess.run(cmd)
    if proc.returncode != 1:
        print()
        print(f"Exitcode {proc.returncode}")
        break
    success += 1
    print(success)

reproducer.py:

import faulthandler
import sys
faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__)
assert False

ZeroIntensity

I think this is a good idea anyway, yeah. We might want to document that faulthandler.dump_traceback_later can sometimes crash during interpreter finalization.

ZeroIntensity · 2025-11-03T15:26:08Z

Include/internal/pycore_interpframe.h

+// Similar to _PyFrame_GetCode(), but return NULL if the frame is invalid or
+// freed. Used by dump_frame() in Python/traceback.c.


It might be worth adding a comment that this isn't completely reliable.

Good idea. I added a sentence for that in the 3 added functions.

vstinner · 2025-11-04T10:48:46Z

Merged, thanks for your reviews!

miss-islington-app · 2025-11-04T10:51:52Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…40921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c) Co-authored-by: Victor Stinner <[email protected]>

bedevere-app · 2025-11-04T10:51:59Z

GH-140981 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2025-11-04T11:16:20Z

GH-140985 is a backport of this pull request to the 3.13 branch.

) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c)

#140981) gh-140815: Fix faulthandler for invalid/freed frame (GH-140921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c) Co-authored-by: Victor Stinner <[email protected]>

…140985) gh-140815: Fix faulthandler for invalid/freed frame (#140921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c)

) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed.

vstinner requested review from ericsnowcurrently and markshannon as code owners November 2, 2025 18:15

bedevere-app bot added the awaiting core review label Nov 2, 2025

bedevere-app bot mentioned this pull request Nov 2, 2025

SEGV in faulthandler.dump_traceback_later #140815

Closed

Add NEWS entry

4059837

vstinner mentioned this pull request Nov 2, 2025

GH-140815: Acquire GIL or stop the world when calling _Py_DumpTracebackThreads from faulthandler_thread #140895

Closed

sergey-miryanov reviewed Nov 2, 2025

View reviewed changes

Include/internal/pycore_interpframe.h Show resolved Hide resolved

sergey-miryanov reviewed Nov 2, 2025

View reviewed changes

Python/traceback.c Outdated Show resolved Hide resolved

picnixz reviewed Nov 2, 2025

View reviewed changes

Include/internal/pycore_interpframe.h Outdated Show resolved Hide resolved

Include/internal/pycore_interpframe.h Show resolved Hide resolved

Include/internal/pycore_interpframe.h Show resolved Hide resolved

Address reviews

9b26187

efimov-mikhail reviewed Nov 2, 2025

View reviewed changes

Include/internal/pycore_code.h Outdated Show resolved Hide resolved

Update Include/internal/pycore_code.h

7d9a4b8

sergey-miryanov approved these changes Nov 2, 2025

View reviewed changes

picnixz reviewed Nov 2, 2025

View reviewed changes

Python/traceback.c Outdated Show resolved Hide resolved

Python/traceback.c Show resolved Hide resolved

ZeroIntensity reviewed Nov 2, 2025

View reviewed changes

vstinner added 3 commits November 3, 2025 12:44

Enhance _PyMem_IsPtrFreed()

c208539

Stricter _PyFrame_SafeGetCode()

c273f99

coding style

3764548

ZeroIntensity approved these changes Nov 3, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Nov 3, 2025

Document that safe functions are not 100% reliable

6239435

vstinner merged commit a84181c into python:main Nov 4, 2025
46 checks passed

vstinner deleted the faulthandler_freed branch November 4, 2025 10:48

bedevere-app bot removed the awaiting merge label Nov 4, 2025

vstinner added the needs backport to 3.14 bugs and security fixes label Nov 4, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Nov 4, 2025

sergey-miryanov mentioned this pull request Nov 5, 2025

Segfault in faulthandler signal handler with threads #116008

Closed

		// Similar to _PyFrame_GetCode(), but return NULL if the frame is invalid or
		// freed. Used by dump_frame() in Python/traceback.c.

Uh oh!

gh-140815: Fix faulthandler for invalid/freed frame #140921

gh-140815: Fix faulthandler for invalid/freed frame #140921

Uh oh!

Conversation

vstinner commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergey-miryanov commented Nov 2, 2025

Uh oh!

sergey-miryanov commented Nov 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner commented Nov 2, 2025

Uh oh!

vstinner commented Nov 2, 2025

Uh oh!

Uh oh!

efimov-mikhail commented Nov 2, 2025

Uh oh!

vstinner commented Nov 2, 2025

Uh oh!

sergey-miryanov left a comment

Choose a reason for hiding this comment

Uh oh!

sergey-miryanov commented Nov 2, 2025

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity Nov 2, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner commented Nov 4, 2025

Uh oh!

miss-islington-app bot commented Nov 4, 2025

Uh oh!

bedevere-app bot commented Nov 4, 2025

Uh oh!

bedevere-app bot commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vstinner commented Nov 2, 2025 •

edited

Loading

vstinner commented Nov 3, 2025 •

edited

Loading