gh-132775: Add _PyCode_VerifyStateless()#133221

ericsnowcurrently · 2025-04-30T18:37:04Z

"Stateless" code is a function or code object which does not rely on external state or internal state. It may rely on arguments and
builtins, but not globals or a closure. I've left a comment in pycore_code.h that provides more detail.

We also add _PyFunction_VerifyStateless(). The new functions will be used in several later changes that facilitate "sharing" functions and code objects between interpreters.

Issue: Improvements to test.support.interpreters.Interpreter #132775

Include/internal/pycore_code.h

Objects/codeobject.c

ericsnowcurrently · 2025-05-05T19:44:16Z

!buildbot s390x RHEL8 LTO + PGO

bedevere-bot · 2025-05-05T19:44:20Z

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit d45c0e0 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133221%2Fmerge

The command will test the builders whose names match following regular expression: s390x RHEL8 LTO \+ PGO

The builders matched are:

s390x RHEL8 LTO + PGO PR

neonene · 2025-05-05T20:50:53Z

Objects/codeobject.c

+
+    // The last instruction either returns or raises.  We can take advantage
+    // of that for a quick exit.
+    _Py_CODEUNIT final = _Py_GetBaseCodeUnit(co, len-1);


Is the warning on Windows fixable on this PR?

nascheme · 2025-05-06T00:37:39Z

This looks to be failing on the iOS buildbot.

test_code (test)
Doctest: test.test_code ... ok
======================================================================
ERROR: test_stateless (test.test_code.CodeTest.test_stateless) [(<function spam_N at 0x10fab5b50>, '(code)')]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/3749308D-3FDE-41E4-B27C-3A4DD1FCF87A/data/Containers/Bundle/Application/40457EBF-BF74-4E58-B5DB-F52F8BC085A9/iOSTestbed.app/python/lib/python3.14/test/test_code.py", line 1049, in test_stateless
    _testinternalcapi.verify_stateless_code(func.__code__)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
ValueError: only basic code objects are supported
======================================================================
ERROR: test_stateless (test.test_code.CodeTest.test_stateless) [(<function spam_C at 0x10fab4a10>, '(code)')]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/3749308D-3FDE-41E4-B27C-3A4DD1FCF87A/data/Containers/Bundle/Application/40457EBF-BF74-4E58-B5DB-F52F8BC085A9/iOSTestbed.app/python/lib/python3.14/test/test_code.py", line 1049, in test_stateless
    _testinternalcapi.verify_stateless_code(func.__code__)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
ValueError: only basic code objects are supported

https://buildbot.python.org/#/builders/1380/builds/3582/

encukou · 2025-05-06T06:08:49Z

The test fails when it's run twice:

./python -m test test_code test_code

encukou · 2025-05-06T06:55:22Z

Looks like setting sys.settrace with opcode tracing causes all code to be instrumented when it's next run.

import sys
import _testinternalcapi

def foo():
    pass

def trace_temporarily():
    sys._getframe().f_trace_opcodes = True
    sys.settrace(print)
    sys.settrace(None)
    sys._getframe().f_trace_opcodes = False
trace_temporarily()

_testinternalcapi.verify_stateless_code(foo.__code__)
foo()
_testinternalcapi.verify_stateless_code(foo.__code__)  # fails

In this case, test_code_equal_with_instrumentation does this. If it runs first, then some test that exepcutes the tested functions (to get the closures I guess), then test_stateless will fail.

encukou · 2025-05-06T07:05:47Z

I'm not sure what exactly "statelessness" is used for in the subinterpreters code. But, it looks like if it is to be useful, it either

needs to be treated as an important property that can't be broken by optimizations, or
the function should be _PyCode_IsStateless, returning a bool, with typical users needing to implement a fallback (copying the code object should be enough?)

Unfortunately, given the timing, I think this needs to be reverted for 3.14 :(

hugovk · 2025-05-06T07:09:33Z

@encukou Please could you open a revert PR?

encukou · 2025-05-06T07:10:14Z

Will do. Resolving conflicts.

…)" This reverts commit d270bb5. Note that the test functions in _code_definitions are not reverted.

…" (#133497)

hugovk · 2025-05-06T10:10:31Z

Reverted in #133497.

ericsnowcurrently · 2025-05-06T13:59:17Z

Thanks for jumping on this. I'll sort it out. FYI, the relevant code is _PyCode_CheckNoInternalState(). It isn't clear to me how test_code_equal_with_instrumentation is affecting the functions used in test_stateless, but it probably doesn't matter. :)

…tateless() (pythongh-133221)" (python#133497)" This reverts commit 3c73cf5.

This reverts commit 3c73cf5 (gh-133497), which itself reverted the original commit d270bb5 (gh-133221). We reverted the original change due to failing android tests. The checks in _PyCode_CheckNoInternalState() were too strict, so we've relaxed them.

The problem we're fixing here is that we were using PyDict_Size() on "defaults", which it is actually a tuple. We're also adding some explicit type checks. This is a follow-up to gh-133221/gh-133528.

The problem we're fixing here is that we were using PyDict_Size() on "defaults", which it is actually a tuple. We're also adding some explicit type checks. This is a follow-up to pythongh-133221/pythongh-133528. (cherry picked from commit dafd141) Co-authored-by: Eric Snow <[email protected]>

gh-132775: Fix _PyFunctIon_VerifyStateless() () The problem we're fixing here is that we were using PyDict_Size() on "defaults", which it is actually a tuple. We're also adding some explicit type checks. This is a follow-up to gh-133221/gh-133528. (cherry picked from commit dafd141, AKA gh-134900) Co-authored-by: Eric Snow <[email protected]>

"Stateless" code is a function or code object which does not rely on external state or internal state. It may rely on arguments and builtins, but not globals or a closure. I've left a comment in pycore_code.h that provides more detail. We also add _PyFunction_VerifyStateless(). The new functions will be used in several later changes that facilitate "sharing" functions and code objects between interpreters.

…() (pythongh-133221)" (python#133497)

…33528) This reverts commit 3c73cf5 (pythongh-133497), which itself reverted the original commit d270bb5 (pythongh-133221). We reverted the original change due to failing android tests. The checks in _PyCode_CheckNoInternalState() were too strict, so we've relaxed them.

The problem we're fixing here is that we were using PyDict_Size() on "defaults", which it is actually a tuple. We're also adding some explicit type checks. This is a follow-up to pythongh-133221/pythongh-133528.

ericsnowcurrently added 5 commits April 30, 2025 12:32

Add _PyCode_CheckPureFunction().

0b2ebdd

Add _PyCode_VerifyStateless().

a27039c

Clarify what "stateless" means.

e15d418

Add tests.

2d79875

Add _PyFunction_VerifyStateless().

8089a33

ericsnowcurrently added the skip news label Apr 30, 2025

ericsnowcurrently requested a review from markshannon as a code owner April 30, 2025 18:37

bedevere-app bot added the awaiting core review label Apr 30, 2025

bedevere-app bot mentioned this pull request Apr 30, 2025

Improvements to test.support.interpreters.Interpreter #132775

Closed

ericsnowcurrently requested a review from iritkatriel April 30, 2025 18:38

iritkatriel reviewed Apr 30, 2025

View reviewed changes

Include/internal/pycore_code.h Outdated Show resolved Hide resolved

Include/internal/pycore_code.h Outdated Show resolved Hide resolved

Objects/codeobject.c Outdated Show resolved Hide resolved

ericsnowcurrently added 2 commits April 30, 2025 19:21

Adjust a comment.

1b67971

Do not return NULL from infallible functions.

a489e13

ZeroIntensity reviewed May 1, 2025

View reviewed changes

Objects/codeobject.c Show resolved Hide resolved

ericsnowcurrently added 4 commits May 1, 2025 18:52

Fix the case where a function only raises.

2a1ae0e

Detect globals even if there were no builtins used.

51ff183

Fix a pointer indirection.

4be26bb

Merge branch 'main' into add-pycode-verify-stateless

d45c0e0

neonene reviewed May 5, 2025

View reviewed changes

Fix a compiler warning.

7bdfcac

ericsnowcurrently enabled auto-merge (squash) May 5, 2025 21:29

ericsnowcurrently merged commit d270bb5 into python:main May 5, 2025
38 checks passed

bedevere-app bot removed the awaiting core review label May 5, 2025

ericsnowcurrently deleted the add-pycode-verify-stateless branch May 5, 2025 21:49

encukou added a commit to encukou/cpython that referenced this pull request May 6, 2025

Revert "pythongh-132775: Add _PyCode_VerifyStateless() (pythongh-133221…

e7d7e26

…)" This reverts commit d270bb5. Note that the test functions in _code_definitions are not reverted.

hugovk pushed a commit that referenced this pull request May 6, 2025

gh-132775: Revert "gh-132775: Add _PyCode_VerifyStateless() (gh-133221)…

3c73cf5

…" (#133497)

ericsnowcurrently mentioned this pull request May 6, 2025

co_instrumentation and co_monitoring Unexpectedly Set #133524

Open

ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this pull request May 6, 2025

Revert "pythongh-132775: Revert "pythongh-132775: Add _PyCode_VerifyS…

eab4c13

…tateless() (pythongh-133221)" (python#133497)" This reverts commit 3c73cf5.

This was referenced May 6, 2025

gh-132775: Unrevert "Add _PyCode_VerifyStateless()" #133528

Merged

[3.14] gh-132775: Unrevert "Add _PyCode_VerifyStateless()" #133625

Merged

ericsnowcurrently mentioned this pull request May 29, 2025

gh-132775: Fix _PyFunctIon_VerifyStateless() #134900

Merged

miss-islington mentioned this pull request May 29, 2025

[3.14] gh-132775: Fix _PyFunctIon_VerifyStateless() (GH-134900) #134901

Merged

Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025

pythongh-132775: Revert "pythongh-132775: Add _PyCode_VerifyStateless…

eec7b86

…() (pythongh-133221)" (python#133497)

Uh oh!

Conversation

ericsnowcurrently commented Apr 30, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ericsnowcurrently commented May 5, 2025

Uh oh!

bedevere-bot commented May 5, 2025

Uh oh!

neonene May 5, 2025

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently May 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nascheme commented May 6, 2025 • edited by hugovk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented May 6, 2025

Uh oh!

encukou commented May 6, 2025 • edited by hugovk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented May 6, 2025

Uh oh!

hugovk commented May 6, 2025

Uh oh!

encukou commented May 6, 2025

Uh oh!

hugovk commented May 6, 2025

Uh oh!

ericsnowcurrently commented May 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ericsnowcurrently commented Apr 30, 2025 •

edited by bedevere-app bot

Loading

nascheme commented May 6, 2025 •

edited by hugovk

Loading

encukou commented May 6, 2025 •

edited by hugovk

Loading