During metaclass.__init__, super() of the constructed class does not work

[tecki]

BPO	23722
Nosy	@ncoghlan, @larryhastings, @ned-deily, @ericsnowcurrently, @serhiy-storchaka, @tecki, @timgraham, @Vgr255, @miss-islington
PRs	bpo-23722: Raise RuntimeError for absent __classcell__. #6931 bpo-23722: Emit a RuntimeWarning for absent __classcell__. #6933 bpo-23722: Fix docs for future __classcell__ changes. #6999 [3.7] bpo-23722: Fix docs for future __classcell__ changes. (GH-6999) #7000 [3.6] bpo-23722: Fix docs for future __classcell__ changes. (GH-6999) #7001
Files	classcell.patch: The patch for the changes described here classcell.patch: the rebased patch issue23722_enhanced_classcell_tests.diff: Rejected idea recorded for design history purposes issue23722_classcell_reference_validation.diff: Revised implementation with stricter self-validation issue23722_documentation_updates.diff: Documentation updates for classcell & PEP 487 hooks issue23722_classcell_reference_validation_v2.diff: Address Serhiy's review comments, includes docs in same patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-20.06:14:17.161>
created_at = <Date 2015-03-20.13:44:21.407>
labels = ['interpreter-core', 'type-bug', '3.8']
title = 'During metaclass.__init__, super() of the constructed class does not work'
updated_at = <Date 2018-05-20.06:14:17.160>
user = 'https://github.com/tecki'

bugs.python.org fields:

activity = <Date 2018-05-20.06:14:17.160>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-20.06:14:17.161>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2015-03-20.13:44:21.407>
creator = 'Martin.Teichmann'
dependencies = []
files = ['43766', '44533', '45736', '45743', '45744', '45748']
hgrepos = []
issue_num = 23722
keywords = ['patch']
message_count = 38.0
messages = ['238672', '238795', '238992', '270642', '270675', '275620', '275652', '275655', '275665', '275697', '275731', '275732', '275733', '282240', '282246', '282248', '282270', '282317', '282318', '282320', '282323', '282324', '282328', '282335', '282336', '282337', '282383', '282387', '282391', '282474', '316900', '316902', '316904', '317153', '317154', '317155', '317156', '317157']
nosy_count = 10.0
nosy_names = ['ncoghlan', 'larry', 'ned.deily', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'Martin.Teichmann', 'Tim.Graham', 'abarry', 'miss-islington']
pr_nums = ['6931', '6933', '6999', '7000', '7001']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23722'
versions = ['Python 3.8']

[tecki]

When a new class is initialized with __init__ in a metaclass,
the __class__ cell of the class about to be initialized is not
set yet, meaning that super() does not work.

This is a known but fixable problem. The attached patch moves
the initialization of __class__ from the end of __build_class__
into type.__new__. This avoids the proliferation of methods
which don't have the __class__ cell set.

[ncoghlan]

I like the change, but even though the current behaviour is arguably buggy (and certainly undesirable) the fix does introduce a new class level attribute that is visible during execution of Python level code.

Perhaps it would be worth rolling this change into PEP-487 and documenting the new transient namespace entry as __classcell__?

[tecki]

A note on the implementation:

The compiler leaves a __cell__ entry in the class' namespace, which
is then filled by type.__new__, and removed from the namespace by
the latter. This is the same way it is done for __qualname__.

As the patch tampers with the compiler, when testing the patch
don't forget to remove old .pyc files, otherwise strange things will
happen.

[tecki]

Currently, a class is created as follows: the compiler turns the class statement into a call to __build_class__. This runs the class body. If __class__ or super() is used within a method of the class, an empty PyCell is created, to be filled later with the class once its done.

The class body returns this cell. Then the metaclass is called to create the actual class, and finally the cell is set to whatever the metaclass returns.

This has the disadvantage that in the metaclasses __new__ and __init__, __class__ and super() are not set. This is a pity, especially because the two parameter version of super() doesn't work either, as the class is not yet bound to a name.

The attached patch lets the compiler add said cell as __classcell__ to the classes namespace, where it will later be taken out by type.__new__ in order to be properly filled.

This resembles the approach used for __qualname__, with the difference that __qualname__ is already added at the beginning of the classes body, such that it is visible to the user.

This way __class__ will be properly set immediately after it is created, thus all methods are immediately usable, already in a metaclasses __new__ or __init__.

This changes the behavior if a metaclass returns another class. currently, __build_class__ will try to set the __class__ in the methods of the class body to whatever __new__ returns, which might be completely unrelated to the classes body.

[gvanrossum]

I don't think this requires adding it to the PEP, and I think doing this is fine. (But I can't review the code.)

[ncoghlan]

Martin, the patch isn't currently applying to trunk - would you have time to take a look at that?

Ned, this is tangentially related to Martin's work on subclass initialization in PEP-487: one of the current problems with zero-argument super is that we don't actually populate the class cell until quite late in the type creation process, so even after the metaclass.__new__ call finishes, zero-argument super still doesn't work yet.

That aspect of the change is clearly a bug fix, but fixing it will have the side-effect of making "__cell__" visible in the class body during execution as a CPython implementation detail.

Would that still be OK to go into beta 2 rather than beta 1?

(Assigned to Ned due to the release management question)

[gvanrossum]

That aspect of the change is clearly a bug fix

I am happy to *rule* that we can treat it as a bugfix, but I disagree
that it's *clearly* a bugfix. It's definitely debatable. This area of
the language is so obscure and so few people remember why it was done
the way that it's done that I expect that someone out there will be
unhappy about the change. But... change happens, so it's okay.

(Please don't respond arguing the "clearly" part, just go ahead and do it. :-)

[ncoghlan]

Now that you point it out, I agree "clearly" is overstating things when it comes to claiming bug fix status for a form of usage that has never worked in the entire life of zero-argument super :)

[tecki]

This is the originial patch rebased such that it applies to the current master.

As a detail in the discussion: "__classcell__" is not visible during the execution of the class body, as it is added at the end of the class body. In this regard, it is different from "__qualname__", which is set at the beginning of the class body such that it may be changed.

The new __classcell__ does show up, however, in the namespace parameter to the __new__ method of the metaclass.

[gvanrossum]

Nick, if you feel like doing this, go ahead, either before or after
beta1 (but if you want to do it before please do it quickly).

(Off-topic: boy do I miss CI that triggers when you send a patch for review...)

On Sat, Sep 10, 2016 at 11:14 AM, Martin Teichmann
<report@bugs.python.org> wrote:

Martin Teichmann added the comment:

This is the originial patch rebased such that it applies to the current master.

As a detail in the discussion: "__classcell__" is not visible during the execution of the class body, as it is added at the end of the class body. In this regard, it is different from "__qualname__", which is set at the beginning of the class body such that it may be changed.

The new __classcell__ does show up, however, in the namespace parameter to the __new__ method of the metaclass.

----------
Added file: http://bugs.python.org/file44533/classcell.patch

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue23722\>

---

[ncoghlan]

Reassigning to myself given Guido's +1