Special method lookup fails on uninitialized types

[ztane]

BPO	26906
Nosy	@gvanrossum, @terryjreedy, @ericvsmith, @serhiy-storchaka, @ztane, @orenmn, @iritkatriel
Files	init_method_descr_types.patch init_types-2.7.patch init_type_in_pytype_lookup.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2021-10-18.21:23:30.922>
created_at = <Date 2016-05-02.13:03:48.822>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Special method lookup fails on uninitialized types'
updated_at = <Date 2021-10-18.21:23:30.921>
user = 'https://github.com/ztane'

bugs.python.org fields:

activity = <Date 2021-10-18.21:23:30.921>
actor = 'iritkatriel'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2021-10-18.21:23:30.922>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2016-05-02.13:03:48.822>
creator = 'ztane'
dependencies = []
files = ['42685', '42716', '42718']
hgrepos = []
issue_num = 26906
keywords = []
message_count = 26.0
messages = ['264647', '264648', '264651', '264730', '264816', '264828', '264840', '264843', '264844', '264912', '264920', '264990', '265009', '265010', '265018', '265023', '265058', '265073', '265081', '277875', '277912', '277913', '278247', '278287', '278290', '404223']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'terry.reedy', 'eric.smith', 'python-dev', 'serhiy.storchaka', 'ztane', 'Oren Milman', 'iritkatriel']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26906'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[ztane]

This is an annoying heisenbug; it seems that some objects cannot be formatted until you explicitly do obj.format. For example object.__reduce__ behaves like this:

    Python 2.7.10 (default, Oct 14 2015, 16:09:02)
    [GCC 5.2.1 20151010] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> format(object.__reduce__)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: Type method_descriptor doesn't define __format__
    >>> format(object.__reduce__)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: Type method_descriptor doesn't define __format__
    >>> object.__reduce__.__format__
    <built-in method __format__ of method_descriptor object at 0x7f67563ed0e0>
    >>> format(object.__reduce__)
    "<method '__reduce__' of 'object' objects>"

I can replicate this in 2.7.9, .10 and .11 on Ubuntu and Debian, though it works on Windows Python, works in 2.6.6, and Pythons 3 wherever I've tried, but I've heard this also failing on Python 3.

[ztane]

s/explicitly do/explicitly access/

[serhiy-storchaka]

Proposed patch makes method descriptors types to be explicitly initialized as in 3.x.

[serhiy-storchaka]

There is similar issue on 3.x:

>>> import array
>>> it = iter(array.array('i'))
>>> format(it)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Type arrayiterator doesn't define __format__
>>> type(it).__format__
<method '__format__' of 'object' objects>
>>> format(it)
'<arrayiterator object at 0xb703f4ec>'

[serhiy-storchaka]

A number of other types are not initialized until you request an attribute.

Here is larger patch for 2.7 that makes 38 types to be explicitly initialized.

[serhiy-storchaka]

An alternative way is just call PyType_Ready from _PyType_Lookup if type->tp_mro is NULL.

Here is a patch against 2.7 that restores the solution from bpo-551412, but returns NULL if type->tp_mro is still NULL after calling PyType_Ready. I found one place in tests when this is happened (CIOTest.test_IOBase_finalize in test_io).

[gvanrossum]

Serhiy, I'm happy to help, but I'm not sure what you're asking me to do. Decide between different patches? I can't even repro the issue.

[serhiy-storchaka]

Added a check for Py_TPFLAGS_READYING to prevent recursive calling.

[serhiy-storchaka]

The problem is that format() fails for instances of some classes, because the type still is not initialized. The simplest example -- list iterator.

>>> format(iter([]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Type listiterator doesn't define __format__

After forcing type initialization (for example by getting any type's attribute), format() becomes working.

>>> type(iter([])).foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'listiterator' has no attribute 'foo'
>>> format(iter([]))
'<listiterator object at 0xb708d0ec>'

I afraid that format() is just one example, and there are other functions or operators that don't work or work incorrectly if the type was not initialized.

init_types-2.7.patch adds explicit initialization of 38 types (I didn't check that all of them need this, but I suppose they do). This is large patch, and I'm not sure that it fixes all types.

Other way is to try to initialize the type just in _PyType_Lookup if it is not initialized. This is simple change, but a comment in _PyType_Lookup warns me. I found that this solution was already applied as an attempt to fix bpo-551412, but then reverted. Since you seem to be the most knowledgeable with this code, I'm asking you what was wrong with this approach and how we can fix this.

Python 3.x also suffers from this bug, but it is reproduced with less types. For example it isn't reproduced for list iterator. I don't know why.

[ztane]

I can reproduce the bug in 3.5.0+ Ubuntu with list iterator, if I execute python with -S:

    % python3.5 -S
    Python 3.5.0+ (default, Oct 11 2015, 09:05:38) 
    [GCC 5.2.1 20151010] on linux
    >>> format(iter([]))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: Type list_iterator doesn't define __format__

Thus here it depends on the stuff that site does or doesn't do. Iterating over a list iterator does *not* trigger the initialization. Printing it doesn't help either, or anything else that does not touch the non-magic attributes. I am not even sure what the site.py and such are doing to the list iterator class to trigger the initialization.

[ztane]

And to the other things failing, I was trying to find out which of the magic method ops fail, and for that tried to find out the dir of list iterator. Well...

    % python3.5 -S          
    Python 3.5.0+ (default, Oct 11 2015, 09:05:38) 
    [GCC 5.2.1 20151010] on linux
    >>> dir(iter([]))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: object does not provide __dir__