__init_subclass__ causes TypeError when used with standard library metaclasses (such as ABCMeta)

[NateSoares]

BPO	29581
Nosy	@ncoghlan, @ambv, @tecki, @ilevkivskyi, @Mariatta
PRs	bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ #527 [3.6] bpo-29581: Make __init_subclass__ work on ABCMeta type classes #1282

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-16.08:17:14.953>
created_at = <Date 2017-02-16.18:54:07.670>
labels = ['type-bug', '3.7']
title = '__init_subclass__ causes TypeError when used with standard library metaclasses (such as ABCMeta)'
updated_at = <Date 2017-06-16.08:17:14.951>
user = 'https://bugs.python.org/NateSoares'

bugs.python.org fields:

activity = <Date 2017-06-16.08:17:14.951>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-16.08:17:14.953>
closer = 'ncoghlan'
components = []
creation = <Date 2017-02-16.18:54:07.670>
creator = 'Nate Soares'
dependencies = []
files = []
hgrepos = []
issue_num = 29581
keywords = []
message_count = 9.0
messages = ['287966', '288119', '288255', '288450', '288451', '288453', '290179', '295312', '296165']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'lukasz.langa', 'Martin.Teichmann', 'levkivskyi', 'Kevin Shweh', 'Mariatta', 'Nate Soares']
pr_nums = ['527', '1282']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29581'
versions = ['Python 3.6', 'Python 3.7']

[NateSoares]

I believe I've found a bug (or, at least, critical shortcoming) in the way that python 3.6's __init_subclass__ interacts with abc.ABCMeta (and, presumably, most other metaclasses in the standard library). In short, if a class subclasses both an abstract class and a class-that-uses-init_subclass, and the __init_subclass__ uses keyword arguments, then this will often lead to TypeErrors (because the metaclass gets confused by the keyword arguments to __new__ that were meant for __init_subclass__).

Here's an example of the failure. This code:

from abc import ABCMeta
class Initifier:
    def __init_subclass__(cls, x=None, **kwargs):
        super().__init_subclass__(**kwargs)
        print('got x', x)

class Abstracted(metaclass=ABCMeta):
    pass

class Thingy(Abstracted, Initifier, x=1):
    pass

thingy = Thingy()

raises this TypeError when run:

Traceback (most recent call last):
  File "<filename>", line 10, in <module>
    class Thingy(Abstracted, Initifier, x=1):
TypeError: __new__() got an unexpected keyword argument 'x'

See http://stackoverflow.com/questions/42281697/typeerror-when-combining-abcmeta-with-init-subclass-in-python-3-6 for further discussion.

[ncoghlan]

This is going to be the case anytime an attempt is made to combine parent classes with incompatible constructor signatures. "type" is unusual in that it goes to great lengths to present an adaptive signature that aligns with whatever the class definition does.

The main relevant trick is to filter the extra arguments out from those passed to metaclasses such that only init_subclass sees them:

================

def ignore_extra_args(base):
    base_meta = type(base)
    class _FilteredMeta(base_meta):
        def __new__(*args, **kwds):
            return base_meta.__new__(*args)
        def __init__(*args, **kwds):
            return base_meta.__init__(*args)
    class _Filtered(base, metaclass=_FilteredMeta):
        pass
    return _Filtered

class InitX():
    def __init_subclass__(cls, x=None):
        print('x')

from abc import ABCMeta
class Abstract(metaclass=ABCMeta):
    pass

class AbstractWithInit(ignore_extra_args(Abstract), InitX, x=1):
    pass

AbstractWithInit()

================

If folks were to iterate on "ignore_extra_args" variants outside the standard library with 3.6, then it would be something we could sensibly standardise for 3.7.

[KevinShweh]

Doesn't that ignore_extra_args thing prevent InitX.__init_subclass__ from receiving the x argument it wanted? It doesn't seem like a solution.

[ncoghlan]

No, the filtering is only applied to the __new__ and __init__ calls on the metaclass, not to the __init_subclass__ call.

Showing the last part of an interactive session where I ran the above commands:

===========

>>> class AbstractWithInit(ignore_extra_args(Abstract), InitX, x=1):
...     pass
... 
x
>>> AbstractWithInit()
<__main__.AbstractWithInit object at 0x7f9086694a58>
>>>

===========

[ncoghlan]

Oops, and now I see the typo in the example code, it seems you're right. I was misremembering an earlier more decorator-like variant of the design where we didn't rely on passing the __init_subclass__ arguments through the metaclass constructor.

[ncoghlan]

Note that the publisher side workaround for this is relatively straightforward: __init_subclass__ implementations that want to be compatible with arbitrary metaclasses will need to take any additional parameters as class attributes (which they may delete), rather than as class header keyword arguments.

For 3.7 though, it probably makes sense to update abc.ABCMeta to pass along arbitrary keyword arguments based on the same rationale as used in PEP-487 to justify making this change for type itself: by default, type.__init_subclass__ will still complain about it, but if someone overrides __init_subclass__ to accept additional keyword arguments, doing so will just work.

[ambv]

New changeset bd583ef by Łukasz Langa (Nate) in branch 'master':
bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ (#527)
bd583ef

[Mariatta]

New changeset 6fb12b5 by Mariatta (Nate) in branch '3.6':
bpo-29581: bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ (GH-527) (GH-1282)
6fb12b5

[ncoghlan]

Thanks for the work on resolving this, all!