gh-142048: Fix quadratically increasing GC delays #142051

kevmo314 · 2025-11-28T20:41:35Z

Fixes quadratically increasing GC delays in the free-threaded build. This is done by updating the local to global allocation count propagation to require the global count to be non-negative. This matches the existing behavior in non-free-threaded gc.c. Additionally, adds a flush that I found was causing allocations to be lost.

Issue: Negative GC gen0 count in free threaded python delaying GC #142048

python-cla-bot · 2025-11-28T20:41:40Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2025-11-28T20:41:40Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

sergey-miryanov · 2025-11-28T21:04:15Z

cc @nascheme @colesbury

kevmo314 · 2025-11-28T23:19:40Z

Thinking about this solution a little more, perhaps an alternative strategy that's more aligned with the original intent would be to only update the global counter to be non-negative, allowing the local ones to be negative. I'm not sure how to do that with the atomics though.

I think I have a better intuition about the core bug though: the compounding happens because garbage collection first resets the global count to zero, then frees objects. Each freed object calls record_deallocation(), driving local counts negative, which eventually get flushed to the global counter. Thus the global counter keeps getting more and more negative.

bedevere-app · 2025-11-30T00:02:41Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Adjust allocation count handling to prevent negative values and update GC state accordingly.

bedevere-app · 2025-12-01T20:23:17Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2025-12-01T20:25:53Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Python/pystate.c

Co-authored-by: Sam Gross <[email protected]>

bedevere-app · 2025-12-01T20:34:18Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

colesbury · 2025-12-01T20:35:52Z

Please add a NEWS blurb. You can either use the blurb-it webapp mentioned above or the blurb command. (I find the blurb command via uvx blurb to be the easiest). It should be in the "Core and Builtins" sections. I think the important thing here is that it fixes quadratic behavior in the GC for free threaded Python, not really whether the count can be negative.

colesbury · 2025-12-01T22:15:22Z

Python/pystate.c

+#ifdef Py_GIL_DISABLED
+    // Flush the thread's local GC allocation count to the global count
+    // before the thread state is deleted, otherwise the count is lost.
+    _Py_atomic_add_int(&tstate->interp->gc.young.count,
+                       (int)tstate_impl->gc.alloc_count);
+    tstate_impl->gc.alloc_count = 0;
+#endif
+


Let's remove this for now. It's not essential for the fix and we can add it back correctly in a subsequent PR.

This is causing a thread sanitizer failure. You can see it in the job if you expand the "Display logs" section:

WARNING: ThreadSanitizer: data race (pid=19505) Atomic write of size 4 at 0x55bb91207c44 by thread T52: #0 _Py_atomic_add_int /home/runner/work/cpython/cpython/./Include/cpython/pyatomic_gcc.h:15:10 (python+0x63520e) (BuildId: c37d32af53b8782ca7323d5cb2febaf42023f6ab) #1 tstate_delete_common /home/runner/work/cpython/cpython/Python/pystate.c:1816:5 (python+0x63520e) #2 _PyThreadState_DeleteCurrent /home/runner/work/cpython/cpython/Python/pystate.c:1893:5 (python+0x635582) (BuildId: c37d32af53b8782ca7323d5cb2febaf42023f6ab) ... Previous write of size 4 at 0x55bb91207c44 by main thread: #0 gc_collect_internal /home/runner/work/cpython/cpython/Python/gc_free_threading.c:2240:33 (python+0x5940bb) (BuildId: c37d32af53b8782ca7323d5cb2febaf42023f6ab) #1 gc_collect_main /home/runner/work/cpython/cpython/Python/gc_free_threading.c:2414:5 (python+0x59199c) (BuildId: c37d32af53b8782ca7323d5cb2febaf42023f6ab) #2 _PyGC_Collect /home/runner/work/cpython/cpython/Python/gc_free_threading.c:2723:12 (python+0x591cb7) (BuildId: c37d32af53b8782ca7323d5cb2febaf42023f6ab) ...

The problem is that gc_collect_internal reads this during a stop the world pause, but this thread is no longer considered active (i.e., we have a "detached" thread state that's getting deleted).

We could move this to PyThreadState_Clear(), but let's just not include it for now. I'd like to keep the critical fix small for backporting and "losing" a few local counts probably doesn't matter.

Makes sense, done!

Removed flushing of thread's local GC allocation count before thread state deletion.

colesbury

Thanks!

miss-islington-app · 2025-12-02T00:04:51Z

Thanks @kevmo314 for the PR, and @colesbury for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

) The GC for the free threaded build would get slower with each collection due to effectively double counting objects freed by the GC. (cherry picked from commit eb89286) Co-authored-by: Kevin Wang <[email protected]>

miss-islington-app · 2025-12-02T00:04:58Z

Sorry, @kevmo314 and @colesbury, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker eb892868b31322d7cf271bc25923e14b1f67ae38 3.13

bedevere-app · 2025-12-02T00:05:02Z

GH-142166 is a backport of this pull request to the 3.14 branch.

…ngh-142051) The GC for the free threaded build would get slower with each collection due to effectively double counting objects freed by the GC. (cherry picked from commit eb89286) Co-authored-by: Kevin Wang <[email protected]>

bedevere-app · 2025-12-02T00:08:37Z

GH-142167 is a backport of this pull request to the 3.13 branch.

…h-142166) The GC for the free threaded build would get slower with each collection due to effectively double counting objects freed by the GC. (cherry picked from commit eb89286) Co-authored-by: Kevin Wang <[email protected]>

…142167) The GC for the free threaded build would get slower with each collection due to effectively double counting objects freed by the GC. (cherry picked from commit eb89286) Co-authored-by: Kevin Wang <[email protected]>

colesbury · 2025-12-02T00:49:57Z

I think the buildbot failure is unrelated. The change only affects the free threaded build and the buildbot isn't for free threaded Python.

) The GC for the free threaded build would get slower with each collection due to effectively double counting objects freed by the GC.

reset allocation counts in gc thread-local buffers

a9edb7b

kevmo314 requested review from ZeroIntensity and ericsnowcurrently as code owners November 28, 2025 20:41

bedevere-app bot added the awaiting review label Nov 28, 2025

bedevere-app bot mentioned this pull request Nov 28, 2025

Negative GC gen0 count in free threaded python delaying GC #142048

Closed

Merge branch 'main' into fix-gc-alloc-count-negative

b1a6c16

Fix allocation count decrement and GC state update

9388518

Adjust allocation count handling to prevent negative values and update GC state accordingly.

kevmo314 changed the title ~~gh-142048: Reset allocation counts in gc thread-local buffers~~ gh-142048: Enforce global allocation count to be non-negative Dec 1, 2025

Fix formatting in gc_free_threading.c

f5e46b5

colesbury requested review from colesbury and nascheme December 1, 2025 20:26

colesbury reviewed Dec 1, 2025

View reviewed changes

Python/pystate.c Outdated Show resolved Hide resolved

Update Python/pystate.c

795e703

Co-authored-by: Sam Gross <[email protected]>

kevmo314 changed the title ~~gh-142048: Enforce global allocation count to be non-negative~~ gh-142048: Fix quadratically increasing GC delays Dec 1, 2025

news and lint

8d01b65

kevmo314 requested a review from colesbury December 1, 2025 20:43

colesbury reviewed Dec 1, 2025

View reviewed changes

kevmo314 added 2 commits December 1, 2025 14:27

Remove local GC allocation count flush in pystate.c

8be49b5

Removed flushing of thread's local GC allocation count before thread state deletion.

Merge branch 'main' into fix-gc-alloc-count-negative

f202991

kevmo314 requested a review from colesbury December 1, 2025 23:40

colesbury approved these changes Dec 1, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Dec 1, 2025

colesbury added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes awaiting merge and removed awaiting merge labels Dec 1, 2025

colesbury merged commit eb89286 into python:main Dec 2, 2025
58 checks passed

bedevere-app bot removed the awaiting merge label Dec 2, 2025

miss-islington-app bot assigned colesbury Dec 2, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Dec 2, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Dec 2, 2025

Uh oh!

gh-142048: Fix quadratically increasing GC delays #142051

gh-142048: Fix quadratically increasing GC delays #142051

Uh oh!

Conversation

kevmo314 commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Nov 28, 2025

Uh oh!

sergey-miryanov commented Nov 28, 2025

Uh oh!

kevmo314 commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Nov 30, 2025

Uh oh!

bedevere-app bot commented Dec 1, 2025

Uh oh!

bedevere-app bot commented Dec 1, 2025

Uh oh!

Uh oh!

bedevere-app bot commented Dec 1, 2025

Uh oh!

colesbury commented Dec 1, 2025

Uh oh!

colesbury Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

kevmo314 Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

colesbury left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Dec 2, 2025

Uh oh!

miss-islington-app bot commented Dec 2, 2025

Uh oh!

bedevere-app bot commented Dec 2, 2025

Uh oh!

bedevere-app bot commented Dec 2, 2025

Uh oh!

This comment was marked as resolved.

colesbury commented Dec 2, 2025

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kevmo314 commented Nov 28, 2025 •

edited

Loading

python-cla-bot bot commented Nov 28, 2025 •

edited

Loading

kevmo314 commented Nov 28, 2025 •

edited

Loading