test_typing is flaky

[bitdancer]

BPO	24635
Nosy	@gvanrossum, @vstinner, @larryhastings, @bitdancer, @ambv, @gvanrossum

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2015-09-04.19:49:21.019>
created_at = <Date 2015-07-15.01:34:05.095>
labels = ['type-bug', 'tests', 'release-blocker']
title = 'test_typing is flaky'
updated_at = <Date 2015-09-04.19:57:35.514>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2015-09-04.19:57:35.514>
actor = 'gvanrossum'
assignee = 'gvanrossum'
closed = True
closed_date = <Date 2015-09-04.19:49:21.019>
closer = 'larry'
components = ['Tests']
creation = <Date 2015-07-15.01:34:05.095>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 24635
keywords = []
message_count = 18.0
messages = ['246750', '246854', '247394', '249720', '249721', '249730', '249731', '249735', '249769', '249785', '249809', '249811', '249812', '249813', '249814', '249816', '249820', '249821']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'vstinner', 'larry', 'r.david.murray', 'lukasz.langa', 'python-dev', 'Guido.van.Rossum']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24635'
versions = ['Python 3.5', 'Python 3.6']

[bitdancer]

See for example:

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Non-Debug%203.x/builds/106

It might be a test order dependency, but see

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11931

where the test passed when re-run at the end.

I'm making this a release blocker because it would be pretty sad to ship a new feature with a flaky test. Larry can of course overrule me on that :)

I note that no one is yet listed as the subject expert for the typing module. I've at least added a line for the module to experts.rst.

[gvanrossum]

You can list me as the expert for typing.py, since I wrote it. :-) (However, until mid August I have limited availability since I'm on vacation.)

This looks indeed like a test order dependency. The three failures are all basic failures where an empty set, dict or list is expected to be an instance of the corresponding ABC (AbstractSet, Mapping, Sequence) defined in typing.py. Those ABCs in turn derive (in a slightly sneaky way) from the corresponding ABCs in collections.abc.

My hunch is that some other test messes with the ABC registration cache and doesn't restore it -- or typing.py's sneaky way of deriving from collections.abc has a subtle bug in it. Maybe something is calling reload(collections.abc)?

I haven't done any research to confirm or deny this theory. It shouldn't be too hard to find the (probably only a handful of) tests that mess with the registration cache.

[larryhastings]

Deferring for beta 4. We should still fix before release. Preferably before RC1 depending on the availability of Certain Persons.

[vstinner]

Is there any progress on this issue? It's still failing randomly:
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.5/builds/221/steps/test/logs/stdio

[vstinner]

======================================================================
FAIL: test_abstractset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 955, in test_abstractset
    assert isinstance(set(), typing.AbstractSet)
AssertionError

======================================================================
FAIL: test_bytestring (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 979, in test_bytestring
    assert isinstance(b'', typing.ByteString)
AssertionError

======================================================================
FAIL: test_container (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 951, in test_container
    assert isinstance([], typing.Container)
AssertionError

======================================================================
FAIL: test_iterable (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 936, in test_iterable
    assert isinstance([], typing.Iterable)
AssertionError

======================================================================
FAIL: test_iterator (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 942, in test_iterator
    assert isinstance(it, typing.Iterator)
AssertionError

======================================================================
FAIL: test_mapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 963, in test_mapping
    assert isinstance({}, typing.Mapping)
AssertionError

======================================================================
FAIL: test_mutablemapping (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 967, in test_mutablemapping
    assert isinstance({}, typing.MutableMapping)
AssertionError

======================================================================
FAIL: test_mutablesequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 975, in test_mutablesequence
    assert isinstance([], typing.MutableSequence)
AssertionError

======================================================================
FAIL: test_mutableset (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 959, in test_mutableset
    assert isinstance(set(), typing.MutableSet)
AssertionError

======================================================================
FAIL: test_sequence (test.test_typing.CollectionsAbcTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.5.bolen-ubuntu/build/Lib/test/test_typing.py", line 971, in test_sequence
    assert isinstance([], typing.Sequence)
AssertionError

---

[larryhastings]

Anything happening with this? We tag 3.5.0rc3 in about 36 hours.

[vstinner]

Anything happening with this? We tag 3.5.0rc3 in about 36 hours.

I believe that it's a bug in test_typing, not in the typing module. So it must not block the release.

[vstinner]

Ok, I found a short script to reproduce the bug:

import typing:
assert isinstance([], typing.Container)
assert isinstance([], typing.Container)

The first check calls typing.GenericMeta.__subclasscheck__(list) which calls abc.ABCMeta.__subclasscheck__(typing.Container, list). The problem is that not only abc.ABCMeta.__subclasscheck__() returns False, but it adds also list to typing.Container._abc_negative_cache.

The next check returns False without calling typing.GenericMeta.__subclasscheck__(list), in fact ABCMeta.__instancecheck__(typing.GenericMeta, list) returns immediatly False because list is in typing.Container._abc_negative_cache.

The problem: GenericMeta calls ABCMeta.__subclasscheck__() which returns False, but ABCMeta.__subclasscheck__() also stores the result "False" in cache, whereas it's not the expected result.

Should we override __instancecheck__ in GenericMeta?

@guido: are you interested to fix this complex metaclass/type issue? I don't have time to fix it right now.

[gvanrossum]

Sorry, this is a bug in typing.py. python/typing#155

It's a subtle bug but the first reporter has done some good research; I will try to find time to fix it today.