Overloading int.__pow__ does not work

[terryjreedy]

BPO	1694663
Nosy	@birkenfeld, @rhettinger, @terryjreedy, @mdickinson, @devdanzin, @serhiy-storchaka
Files	mixing_slots.diff: patch against trunk revision 54690 slot_inheritance.diff: An old patch with a few more fixes time_slot_inheritance.py: The timeit script from the last message

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-01-04.18:12:47.497>
created_at = <Date 2007-04-04.23:44:52.000>
labels = ['extension-modules', 'type-bug']
title = 'Overloading  int.__pow__ does not work'
updated_at = <Date 2013-01-05.12:35:24.705>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2013-01-05.12:35:24.705>
actor = 'mark.dickinson'
assignee = 'zseil'
closed = True
closed_date = <Date 2013-01-04.18:12:47.497>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2007-04-04.23:44:52.000>
creator = 'terry.reedy'
dependencies = []
files = ['2342', '12028', '12029']
hgrepos = []
issue_num = 1694663
keywords = ['patch']
message_count = 10.0
messages = ['31702', '31703', '31704', '31705', '75934', '75957', '86635', '110577', '179061', '179119']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'rhettinger', 'terry.reedy', 'mark.dickinson', 'zseil', 'ajaksu2', 'BreamoreBoy', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1694663'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[terryjreedy]

From c.l.p: in response to glitch report, 2nd person showed that it is specific to __pow__

>>> class MyInt(int):
...     __sub__ = int.__add__
# similar lines for 9 other methods omitted
...     __or__ = int.__add__
...     __pow__ = int.__add__
...
>>> i = MyInt(42)
>>> i + 3
45
>>> i - 3
45
# similar outputs omitted
>>> i | 3
45
>>> i ** 3
74088

Another person (3rd) confirmed presence in 2.5
Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)]

[zseil]

Here is a patch (with tests) that should fix this.
There was another problem when the slot wrapper
came from a different type:

>>> class MyInt(int):
...     __mul__ = float.__add__
...
>>> MyInt(3) * 3
9

This now raises a TypeError.

File Added: mixing_slots.diff

[rhettinger]

I'll take a look at your patch.

The root problem is that the wrapper functions will return NULL if they are fed the wrong number of arguments (i.e. the int.__add__ wrapper expects exactly two arguments but the ** call provides three) -- you will see a similar result if int.__neg__ or int.__invert__ are assigned to __add__. It looks like an upstream step is seeing the NULL and deciding that it needs to look to skip the given method call and instead try the base the class.

Am lowering the priority because there is no good use case for deliberate argument mismatches in the override. Am leaving the bug open because we want the appropriate error message to surface.

[zseil]

Hi Raymond,

The problem is not in conversion functions, but in
slot inheritance. If you run the example under a
debugger you can see that

MyInt->tp_as_number->nb_power == int->->tp_as_number->nb_power.

This happens because update_one_slot() doesn't fall
back to the generic slot function if a SlotWrapper in
type's dict has a different conversion function or if
it comes from an unrelated type. The patch simply falls
back to the generic slot function in this case.

[rhettinger]

Georg, do you want to take a look at this patch. I've neglected it
because I don't see why the argument signature matters (neg and
__pow__ take a different number of arguments than the other magic
numeric methods that work just fine).

[zseil]

Hi Raymond,

The signature matters because the current code in update_one_slot()
forgets to set the use_generic flag when slots have different wrappers.
This causes that the slot from the base class is left in the new type.
Slots have different wrappers when their signature differs.

I also found an old patch for this issue that fixes a few more corner
cases, but I can't remember if it is finished and it doesn't apply to
current trunk. I'll try to fix that soon, but here is the original
message and patch:

"""
Here is a new version of the patch. It fixes another bug and
restores the optimization that was lost when dict.__getitem__
and list.__getitem__ became methods with a METH_COEXIST flag.

The bug that this patch fixes shows when a type has the wrong
__new__ method:

>>> class C(object):
...     __new__ = int.__new__
...
>>> C()            # should raise a TypeError, but doesn't
>>> C.__new__(C)   # raises a TypeError, as expected

This means that Guido should buy me a beer :).

Adding __getitem__ to dict's and list's methods slowed down the
corresponding operation for subclasses that don't overwrite
__getitem__. update_one_slot() installs a generic wrapper when
a __special__ method is not a WrapperDescriptor. As a consequence,
this code is slower:

>>> class MyList(list):
...     pass
...
>>> L = MyList([0])
>>> L[0]           # currently slower than without METH_COEXIST

set.__contains__ and dict.__contains__ have the same problem.
I'll attach a timeit script that shows the difference.

The basic idea of this patch is that when all of the __special__
methods for a particular slot are inherited from the same base,
we can safely use that base's slot for the subtype. The patch
compares __special__ methods by identity, which eliminates all
of the problems listed in this bug report. For more details,
see the comments in the patch.

The patch contains only the tests for the bugs reported here;
the old behavior is already thoroughly tested in test_descr's
dynamics(), overloading(), subclasspropagation() and other
functions.

The patch introduces a slight incompatibility with Python 2.4 and
2.5; code that calls PySequence_GetItem(dict_subclass, index) for
dict subclasses can now fail, because tp_as_sequence->sq_item gets
filled only if the subclass overwrites the __getitem__ method.

Here are the results of running the script on a trunk build:

list test: 0.585701534692
dict test: 0.562311969817
set test: 0.468992008761
class test: 0.235781083909

and results of running the script with this patch applied:

list test: 0.167487487935
dict test: 0.149341885631
set test: 0.153591029027
class test: 0.211393347479
"""