OrderedDict has strange behaviour when dict.__setitem__ is used.

[markshannon]

BPO	24726
Nosy	@rhettinger, @terryjreedy, @markshannon, @ericsnowcurrently, @serhiy-storchaka, @mpaolini
Files	test.py tem2.py odict_repr_after_dict_setitem_delitem.patch odict_delitem_iter_hung.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-29.09:39:06.686>
created_at = <Date 2015-07-26.10:20:52.176>
labels = ['extension-modules', 'type-crash']
title = 'OrderedDict has strange behaviour when dict.__setitem__ is used.'
updated_at = <Date 2019-08-29.09:39:06.685>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2019-08-29.09:39:06.685>
actor = 'rhettinger'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-29.09:39:06.686>
closer = 'rhettinger'
components = ['Extension Modules']
creation = <Date 2015-07-26.10:20:52.176>
creator = 'Mark.Shannon'
dependencies = []
files = ['40028', '40089', '40834', '41398']
hgrepos = []
issue_num = 24726
keywords = ['patch']
message_count = 16.0
messages = ['247421', '247426', '247781', '247782', '247792', '253294', '253310', '254045', '254064', '254069', '254071', '254174', '254178', '256912', '350238', '350258']
nosy_count = 7.0
nosy_names = ['rhettinger', 'terry.reedy', 'Mark.Shannon', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'mpaolini']
pr_nums = []
priority = 'low'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue24726'
versions = ['Python 3.5', 'Python 3.6']

[markshannon]

Setting an item in an ordered dict via dict.__setitem__, or by using it as an object dictionary and setting an attribute on that object, creates a dictionary whose repr is:

OrderedDict([<NULL>])

Test case attached.

[mpaolini]

Linking related issues http://bugs.python.org/issue24721 http://bugs.python.org/issue24667 and http://bugs.python.org/issue24685

[terryjreedy]

Attached revised file that runs to completion on 2.7 and 3.x.

[terryjreedy]

Marco, #-prefixed issue numbers like this, bpo-24721, bpo-24667, and bpo-24685, are easier to read.

[rhettinger]

There is a bug in _PyObject_GenericSetAttrWithDict() Objects/object.c where a calls are made to PyDict_SetItem() and PyDict_DelItem() without checking first checking for PyDict_CheckExact().

• In PEP-372, OrderedDict was consciously specified to be a subclass of regular dicts in order to improve substitutability for dicts in most existing code. That decision had some negative consequences as well. It is unavoidable the someone can call the parent class directly and undermine the invariants of the subclass (that is a fact of life for all subclasses that maintain their own state while trying to stay in-sync with state in the parent class -- see http://bugs.python.org/msg247358 for an example).

With pure python code for the subclass, we say, "don't do that". I'll add a note to that effect in the docs for the OD (that said, it is a general rule that applies to all subclasses that have to stay synchronized to state in the parent).

In C version of the OD subclass, we still can't avoid being bypassed (see http://bugs.python.org/issue10977) and having our subclass invariants violated. Though the C code can't prevent the invariants from being scrambled it does have an obligation to not segfault and to not leak something like "OrderedDict([<NULL>])". Ideally, if is possible to detect an invalid state (i.e. the linked link being out of sync with the inherited dict), then a RuntimeError or somesuch should be raised.

[ericsnowcurrently]

FTR, this will likely involve more than just fixing odict_repr().

[serhiy-storchaka]

__repr__() allocates a list with the size len(od) and fills it iterating linked list. If the size of linked list is less then the size of the dict, the rest of the list is not initialized.

Even worse things happened when the size of linked list is greater then the size of the dict. Following example causes a crash:

from collections import OrderedDict
od = OrderedDict()
class K(str):
    def __hash__(self):
        return 1

od[K('a')] = 1
od[K('b')] = 2
print(len(od), len(list(od)))
K.__eq__ = lambda self, other: True
dict.__delitem__(od, K('a'))
print(len(od), len(list(od)))
print(repr(od))

Proposed patch fixes both issues.

[serhiy-storchaka]

Ping.