refleak in typing.py

[1st1]

BPO	28649
Nosy	@gvanrossum, @ned-deily, @serhiy-storchaka, @1st1, @MojoVampire, @ilevkivskyi
Files	slots_gc.patch typing-clear-caches.patch fix-typing-test.diff fix-typing-test-v2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-11-10.16:32:47.208>
created_at = <Date 2016-11-09.16:00:08.573>
labels = ['3.7', 'type-bug', 'library', 'release-blocker']
title = 'refleak in typing.py'
updated_at = <Date 2016-11-10.16:45:07.923>
user = 'https://github.com/1st1'

bugs.python.org fields:

activity = <Date 2016-11-10.16:45:07.923>
actor = 'levkivskyi'
assignee = 'none'
closed = True
closed_date = <Date 2016-11-10.16:32:47.208>
closer = 'yselivanov'
components = ['Library (Lib)']
creation = <Date 2016-11-09.16:00:08.573>
creator = 'yselivanov'
dependencies = []
files = ['45410', '45414', '45415', '45416']
hgrepos = []
issue_num = 28649
keywords = ['patch']
message_count = 61.0
messages = ['280407', '280411', '280412', '280413', '280414', '280415', '280417', '280420', '280421', '280424', '280425', '280426', '280428', '280430', '280431', '280434', '280436', '280437', '280438', '280439', '280440', '280441', '280442', '280443', '280444', '280446', '280447', '280454', '280455', '280457', '280458', '280459', '280460', '280461', '280462', '280463', '280464', '280465', '280466', '280467', '280469', '280481', '280482', '280483', '280484', '280485', '280486', '280487', '280488', '280489', '280493', '280496', '280501', '280513', '280514', '280518', '280519', '280520', '280521', '280523', '280524']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'ned.deily', 'python-dev', 'serhiy.storchaka', 'yselivanov', 'josh.r', 'levkivskyi']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28649'
versions = ['Python 3.6', 'Python 3.7']

[1st1]

Looks like we have a refleak somewhere in the system that typing.py is exposing.

The following code exposes the refleak:

    def test_refleak(self):
        T = typing.TypeVar('T')
        class C(typing.Generic[T], typing.Mapping[int, str]): ...

The change that first triggered the leak is 8f0df4db2b06.

I'm not an expert in typing.py, but I suspect that the real bug must be somewhere in CPython core.

Guido, do you have any idea how 8f0df4db2b06 could trigger this?

[serhiy-storchaka]

Could it be related to leaks in objects with empty __slots__? See bpo-24379 for details.

[1st1]

Could it be related to leaks in objects with empty __slots__? See bpo-24379 for details.

Not sure... Do we have a test/patch for objects-with-empty-slots bug?

[gvanrossum]

Ivan, this was an early typing.py update (merged into CPython on Sept. 27). But the refleak is still with us. Do you have any intuition on what could be going on here?

My own intuition here is lacking, other than thinking there's a lot of metaclass magic going on in this code...

[serhiy-storchaka]

>>> import gc
>>> gc.collect()
23
>>> gc.collect()
0
>>> class A:
...     __slots__ = ()
... 
>>> del A
>>> gc.collect()
2
^ leak

[serhiy-storchaka]

See also https://mail.python.org/pipermail/python-dev/2015-October/141993.html.

[gvanrossum]

Serhiy, why is gc.collect() returning 2 after del A proof of a leak?
Yes, there's a cycle, but it's being GC'ed -- isn't that fine? Without the
slots the collect() call returns a larger number.

[1st1]

Attached patch (slots_gc.patch) makes objects with empty slots tracked by GC:

   class A: __slots__ = ()
   gc.is_tracked(A()) == False  # before
   gc.is_tracked(A()) == True   # with the patch

This doesn't fix the refleak though.

[1st1]

I removed all slots from typing.py, the refleak is still there.

[1st1]

Wow. In default CPython branch, with no modifications, test_typing.py simply crashes in debug mode:

yury@ysmbp ~/dev/py/cpython $ ./python.exe -m test -R3:3 test_typing
Run tests sequentially
0:00:00 [1/1] test_typing
beginning 6 repetitions
123456
.test test_typing failed -- Traceback (most recent call last):
  File "/Users/yury/dev/py/cpython/Lib/test/test_typing.py", line 739, in test_generic_forvard_ref
    self.assertEqual(get_type_hints(foobar, globals(), locals()), {'x': List[List[T]]})
AssertionError: {'x': typing.List[typing.List[~T]]} != {'x': typing.List[typing.List[~T]]}
  {'x': typing.List[typing.List[~T]]}

test_typing failed

1 test failed:
test_typing

Total duration: 1 sec
Tests result: FAILURE

[1st1]

The problem is that functools.lru_cache is used in _tp_cache. If I remove type caching, the original "refleak" is fixed.