PyEval_EvalCodeEx() can no longer be called with code which has (CO_NEWLOCALS | CO_OPTIMIZED) flags

[vstinner]

BPO	43541
Nosy	@vstinner, @markshannon, @pablogsal
PRs	bpo-43541: Fix PyEval_EvalCodeEx() regression #24918

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-18.13:52:01.583>
created_at = <Date 2021-03-18.11:11:13.990>
labels = ['expert-C-API', '3.10']
title = 'PyEval_EvalCodeEx() can no longer be called with code which has (CO_NEWLOCALS | CO_OPTIMIZED) flags'
updated_at = <Date 2021-03-18.13:52:01.583>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-03-18.13:52:01.583>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-18.13:52:01.583>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-03-18.11:11:13.990>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43541
keywords = ['patch']
message_count = 4.0
messages = ['389009', '389011', '389013', '389015']
nosy_count = 3.0
nosy_names = ['vstinner', 'Mark.Shannon', 'pablogsal']
pr_nums = ['24918']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43541'
versions = ['Python 3.10']

[vstinner]

Cython generates a __Pyx_PyFunction_FastCallDict() function which calls PyEval_EvalCodeEx(). With Python 3.9, it worked well. With Python 3.10 in debug mode, it fails with an assertion error:

python3.10: Python/ceval.c:5148: PyEval_EvalCodeEx: Assertion `(((PyCodeObject *)_co)->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == 0' failed.

With Python 3.10 in release mode, it does crash.

Context of the failed assertion:

• Assertion added recently to CPython 3.10 by 0332e56
• The code object flags = (CO_NEWLOCALS | CO_OPTIMIZED | CO_NOFREE)
• Code co_argcount = 2
• Code co_kwonlyargcount = 0
• Cython __Pyx_PyFunction_FastCallDict() called with: nargs=1 and kwargs=NULL

See the Cython issue to a reproducer: cython/cython#4025 (comment)

In Python 3.9, _PyFunction_Vectorcall() has the following fast-path:

    if (co->co_kwonlyargcount == 0 && nkwargs == 0 &&
        (co->co_flags & ~PyCF_MASK) == (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE))
    {
        if (argdefs == NULL && co->co_argcount == nargs) {
            return function_code_fastcall(tstate, co, stack, nargs, globals);
        }
        else if (nargs == 0 && argdefs != NULL
                 && co->co_argcount == PyTuple_GET_SIZE(argdefs)) {
            /* function called with no arguments, but all parameters have
               a default value: use default values as arguments .*/
            stack = _PyTuple_ITEMS(argdefs);
            return function_code_fastcall(tstate, co,
                                          stack, PyTuple_GET_SIZE(argdefs),
                                          globals);
        }
    }

When the bug occurs, __Pyx_PyFunction_FastCallDict() doesn't take the fast-path because nargs < co_argcount (1 < 2).

In Python 3.10, _PyFunction_Vectorcall() is very different:

    if (((PyCodeObject *)f->fc_code)->co_flags & CO_OPTIMIZED) {
        return _PyEval_Vector(tstate, f, NULL, stack, nargs, kwnames);
    }
    else {
        return _PyEval_Vector(tstate, f, f->fc_globals, stack, nargs, kwnames);
    }

PyEval_EvalCodeEx() must not crash if the code object has (CO_NEWLOCALS | CO_OPTIMIZED | CO_NOFREE) flags.

[vstinner]

If I the comment the following assertion in PyEval_EvalCodeEx():

assert ((((PyCodeObject *)_co)->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == 0);

The garbage collector fails while visiting builtins of the function with func_traverse(), at line:

   Py_VISIT(f->func_builtins);

I guess that the problem is that f->func_builtins doesn't hold a strong reference to builtins.

Error:

Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address : 0x7fffea69d650
object refcount : 2265
object type : 0x870e80
object type name: dict
object repr : {'name': 'builtins', 'doc': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the nil' object; Ellipsis represents ...' in slices.", 'package': '', 'loader': <class '_frozen_importlib.BuiltinImporter'>, 'spec': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built-in'), 'build_class': , 'import': , 'abs': , 'all': , 'any': , 'ascii': , ...}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007ffff7c20740 (most recent call first):
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 339 in __init__
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 1953 in __init__
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 2066 in __init__
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 1549 in declare_c_class
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Nodes.py", line 4782 in analyse_declarations
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Nodes.py", line 431 in analyse_declarations
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/ModuleNode.py", line 124 in analyse_declarations
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1608 in visit_ModuleNode
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1598 in __call__
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Pipeline.py", line 335 in run
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Pipeline.py", line 355 in run_pipeline
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 515 in run_pipeline
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 727 in compile_single
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1208 in cythonize_one
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1102 in cythonize
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Cythonize.py", line 97 in cython_compile
File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Cythonize.py", line 223 in main
File "/home/vstinner/dev/numpy/env/bin/cythonize", line 33 in <module>

Extension modules: Cython.Plex.Actions, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.Tempita._tempita, Cython.Compiler.Visitor, Cython.Compiler.FlowControl (total: 6)

Program received signal SIGABRT, Aborted.

[vstinner]

In commit 46496f9, I modified _PyEval_BuiltinsFromGlobals() to return a borrowed reference rather than a strong reference. It seems like I forgot to remove a Py_DECREF() in PyEval_EvalCodeEx()!

I'm working on a fix.

[vstinner]

New changeset fc980e0 by Victor Stinner in branch 'master':
bpo-43541: Fix PyEval_EvalCodeEx() regression (GH-24918)
fc980e0