Zero argument super is broken in 3.6 for methods with a hacked __class__ cell

[mr-nfamous]

BPO	32176
Nosy	@ncoghlan, @mr-nfamous
PRs	bpo-32176: Set CO_NOFREE in the code object constructor #4675 [3.6] bpo-32176: Set CO_NOFREE in the code object constructor (GH-4675) #4684

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-03.13:36:10.122>
created_at = <Date 2017-11-30.07:39:34.993>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Zero argument super is broken in 3.6 for methods with a hacked __class__ cell'
updated_at = <Date 2017-12-03.13:36:10.120>
user = 'https://github.com/mr-nfamous'

bugs.python.org fields:

activity = <Date 2017-12-03.13:36:10.120>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-03.13:36:10.122>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2017-11-30.07:39:34.993>
creator = 'bup'
dependencies = []
files = []
hgrepos = []
issue_num = 32176
keywords = ['patch']
message_count = 7.0
messages = ['307279', '307352', '307414', '307420', '307480', '307511', '307512']
nosy_count = 2.0
nosy_names = ['ncoghlan', 'bup']
pr_nums = ['4675', '4684']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32176'
versions = ['Python 3.6', 'Python 3.7']

[mr-nfamous]

The following code works in 3.3, 3.4, and 3.5, but in 3.6 it throws RuntimeError: super(): bad __class__ cell.

from types import FunctionType, CodeType

def create_closure(__class__):
    return (lambda: __class__).__closure__

def new_code(c_or_f):
    '''A new code object with a __class__ cell added to freevars'''
    c = c_or_f.__code__ if isinstance(c_or_f, FunctionType) else c_or_f
    return CodeType(
        c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
        c.co_stacksize, c.co_flags, c.co_code, c.co_consts, c.co_names,
        c.co_varnames, c.co_filename, c.co_name, c.co_firstlineno,
        c.co_lnotab, c.co_freevars + ('__class__',), c.co_cellvars)

def add_foreign_method(cls, f):
    code = new_code(f.__code__)
    name = f.__name__
    defaults = f.__defaults__
    closure = (f.__closure__ or ()) + create_closure(cls)
    setattr(cls, name, FunctionType(code, globals(), name, defaults, closure))

class List(list):
    def append(self, elem):
        super().append(elem)
    def extend(self, elems):
        super().extend(elems)
def __getitem__(self, i):
    print('foreign getitem')
    return super().__getitem__(i)

add_foreign_method(List, __getitem__)

self = List([1,2,3])
self[0]

[mr-nfamous]

The hacked cell object using this method appears to be changed to NULL when accessed by frame.f_localsplus. I don't know C well enough to find out what's happening because nothing looks different to me in PyFrame_FastToLocalsWithError.

Also creating a closure with:
from ctypes import pythonapi, py_object

new_cell = pythonapi.PyCell_New
new_cell.argtypes = (py_object, )
new_cell.restype = py_object

doesn't solve this either.

[mr-nfamous]

So while CO_NOFREE is set in all versions with the example code, it appears only python 3.6 recognizes that flag and disallows the accessing of the __class__ cell. In this case the error message is bad because it implies that the __class__ cell is the wrong type. Disabling the flag when creating the code objects allows the above code to work.

[ncoghlan]

Given that, I'd say the way to cleanest way to fix this would be to remove these lines from "compute_code_flags" in compile.c:

if (!PyDict_GET_SIZE(c->u->u_freevars) &&
    !PyDict_GET_SIZE(c->u->u_cellvars)) {
    flags |= CO_NOFREE;
}

and replace them with a check like the following in PyCode_New just after we ensure the Unicode string for the filename is ready:

if (!PyTuple_GET_SIZE(freevars) &&
    !PyTuple_GET_SIZE(cellvars)) {
    flags |= CO_NOFREE;
}

That way CO_NOFREE will be set only when appropriate regardless of how the code object is created, rather than relying on the caller to set it correctly.

[ncoghlan]

New changeset 078f181 by Nick Coghlan in branch 'master':
bpo-32176: Set CO_NOFREE in the code object constructor (GH-4675)
078f181

[ncoghlan]

New changeset c8f32aa by Nick Coghlan in branch '3.6':
[3.6] bpo-32176: Set CO_NOFREE in the code object constructor (GH-4684)
c8f32aa

[ncoghlan]

Thanks for the issue report! The fix will be released in 3.6.4 and 3.7.0a3 (both of which are expected to be later this month).