IDLE: test_browser is failed when run twice

[serhiy-storchaka]

BPO	31559
Nosy	@terryjreedy, @serhiy-storchaka
PRs	bpo-31559: Remove test order dependence in idle_test.test_browser. #3708 [3.6] bpo-31559: Remove test order dependence in idle_test.test_browser. (GH-3708) #3709

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2017-09-23.19:24:53.939>
created_at = <Date 2017-09-23.12:06:23.493>
labels = ['3.7', 'expert-IDLE', 'type-bug', 'tests']
title = 'IDLE: test_browser is failed when run twice'
updated_at = <Date 2017-09-23.19:24:53.938>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-09-23.19:24:53.938>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2017-09-23.19:24:53.939>
closer = 'terry.reedy'
components = ['IDLE', 'Tests']
creation = <Date 2017-09-23.12:06:23.493>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31559
keywords = ['patch']
message_count = 5.0
messages = ['302786', '302794', '302795', '302796', '302800']
nosy_count = 2.0
nosy_names = ['terry.reedy', 'serhiy.storchaka']
pr_nums = ['3708', '3709']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31559'
versions = ['Python 3.6', 'Python 3.7']

[serhiy-storchaka]

Buildbots failures:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/101/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x/builds/116/steps/test/logs/stdio

I don't know why test_browser fails first time, but this provokes running it second time why always fails.

$ ./python -m test -vuall -m test_browser test_idle test_idle
...

======================================================================
FAIL: test_gettext (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 181, in test_gettext
    self.assertEqual(self.cbt_C1.GetText(), 'class C1')
AssertionError: 'class C1()' != 'class C1'
- class C1()
?         --
+ class C1

======================================================================
FAIL: test_init (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 175, in test_init
    eq(self.cbt_C1.name, 'C1')
AssertionError: 'C1()' != 'C1'
- C1()
?   --
+ C1

======================================================================
FAIL: test_getsublist (idlelib.idle_test.test_browser.ModuleBrowserTreeItemTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 141, in test_getsublist
    self.assertEqual(sub1.name, 'C0')
AssertionError: 'C0(base)' != 'C0'
- C0(base)
+ C0

======================================================================
FAIL: test_nested (idlelib.idle_test.test_browser.NestedChildrenTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 238, in test_nested
    self.assertEqual(actual_names, expected_names)
AssertionError: Lists differ: ['f0', 'C0(base)', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3'] != ['f0', 'C0', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3']

First differing element 1:
'C0(base)'
'C0'

• ['f0', 'C0(base)', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3']
  ? ------

+ ['f0', 'C0', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3']

----------------------------------------------------------------------

The test is passed if run it once on my computer.

[terryjreedy]

browser.transform_children, when called the first time on a dict of children nodes, rebinds the name attribute of Classes with bases*. Tests that called the real browser.transform_children the first time on a node mutate the global mock_pyclbr_tree. When the tests are re-run in the same process, tests initially run before the mutation are also run after the mutation.

*This is not an issue in current real use. IDLE gets a new tree each time a module browser is opened. The current tree widget calls GetSublist and hence transform_children at most once per node as it caches the result for each node. But don't know what ttk.Treeview does and do know that tests can make multiple calls.

Restoring the tree with
def tearDownModule():
C0.name = 'C0'
C1.name = 'C1'
fixes the failures found when rerunning the test module in the same process, but not the underlying problem of order dependence.

Instead, I added immediate calls to transform_children so that the mock tree is never mutated thereafter by any of the tests. The subsequent class checks that the initial name replacements are correct. All tests see the tree in the same state, so there should not be any order dependence left.

Testing global function calls in a testcase is slightly awkward, but the testcase follows immediately, and making the tree and its nodes global simplifies the tests that use the tree.

I checked that running test_pyclbr and test_idle twice each succeeds.

[terryjreedy]

I don't know why test_browser fails first time,

On both buildbots, the initial rerun passes:
Ran 371 tests in 12.265s OK (skipped=3) Gentoo
Ran 371 tests in 12.047s OK (skipped=1) Windows8

[terryjreedy]

New changeset 99167f8 by Terry Jan Reedy in branch 'master':
bpo-31559: Remove test order dependence in idle_test.test_browser. (bpo-3708)
99167f8

[terryjreedy]

New changeset 429b3b1 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31559: Remove test order dependence in idle_test.test_browser. (GH-3708) (bpo-3709)
429b3b1