some test_idle tests are not re-runnable, producing false failures with regrtest -w option

[ned-deily]

BPO	27714
Nosy	@terryjreedy, @ned-deily, @roseman, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2016-08-21.01:10:27.074>
created_at = <Date 2016-08-08.23:14:56.832>
labels = ['expert-IDLE', 'type-bug']
title = 'some test_idle tests are not re-runnable, producing false failures with regrtest -w option'
updated_at = <Date 2016-08-21.04:21:40.400>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2016-08-21.04:21:40.400>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2016-08-21.01:10:27.074>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2016-08-08.23:14:56.832>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = []
issue_num = 27714
keywords = []
message_count = 20.0
messages = ['272200', '272219', '272222', '272349', '272361', '272362', '272369', '272378', '272379', '272387', '272388', '272567', '272805', '272816', '272819', '272840', '273010', '273257', '273263', '273264']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'ned.deily', 'markroseman', 'python-dev', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27714'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[ned-deily]

If you run test_idle using the standard Python regression test runner, regrtest, and use regrtest's -w option (to re-run failure test verbosely) *without* using regrtest's -j option (to run tests in separate processes), any real test failure triggering a rerun will cause a number of additional false IDLE test case failures because some test cases modify global state in such a way as to be not re-runable.

An example: at the moment there is a real IDLE test case failure (see bpo-27830). If I run test_idle with both -w and -j options:

$ /usr/local/bin/python3.6 -m test -w -j2 -uall test_idle
Run tests in parallel using 2 child processes
0:00:01 [1/1/1] test_idle failed
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

('__test__',

• 'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
  ? -------

+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
1 test failed:
test_idle
Re-running failed tests in verbose mode
Re-running test 'test_idle' in verbose mode
test_autocomplete_event (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
test_delayed_open_completions (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
[...]
test_shell_show (idlelib.idle_test.test_warning.ShellWarnTest) ... ok
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok

======================================================================
FAIL: test_click_help_source (idlelib.idle_test.test_query.HelpsourceGuiTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

('__test__',

• 'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
  ? -------

+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

----------------------------------------------------------------------
Ran 219 tests in 0.977s

FAILED (failures=1)

only the 1 valid test case failure shows up.

But if I run test_idle with just -w (no -j) options:

$ /usr/local/bin/python3.6 -m test -w -uall test_idle
Run tests sequentially
0:00:00 [1/1] test_idle
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

('__test__',

• 'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
  ? -------

+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

test_idle failed
1 test failed:
test_idle
Re-running failed tests in verbose mode
Re-running test 'test_idle' in verbose mode
test_autocomplete_event (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
test_delayed_open_completions (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
[...]
test_no_delete (idlelib.idle_test.test_text.TkTextTest) ... ok
test_init_modal (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_init_nonmodal (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_ok (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_view_file (idlelib.idle_test.test_textview.ViewFunctionTest) ... ok
test_view_text (idlelib.idle_test.test_textview.ViewFunctionTest) ... ok
ERROR
test_init (idlelib.idle_test.test_tree.TreeTest) ... ok
[...]
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok

======================================================================
ERROR: test_init_modal (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: test_init_nonmodal (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: test_ok (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: tearDownModule (idlelib.idle_test.test_textview)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 26, in tearDownModule
    del TV
NameError: name 'TV' is not defined

======================================================================
FAIL: test_click_help_source (idlelib.idle_test.test_query.HelpsourceGuiTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

('__test__',

• 'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
  ? -------

+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

----------------------------------------------------------------------
Ran 219 tests in 0.998s

FAILED (failures=1, errors=4)

now there are 4 or 5 errors in addition to the 1 failure.

The fifth false error is somewhat timing dependent, e.g. it shows up some of the time while the other 4 always show up:

======================================================================
ERROR: setUpClass (idlelib.idle_test.test_autocomplete.AutoCompleteTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
    macosx.setupApp(cls.root, None)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
    overrideRootMenu(root, flist)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
    del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of range

It's not a critical problem but it can be very confusing if you have not run into the problem before and can cause lost time trying to track down the false errors.

[terryjreedy]

Mark, see questions about Mac above. Basically, do we still need to add 'file://'?

The test_textview repeat errors are eliminated by removing 'del TV' in tearDownModule. I added it in what seems to be an excess of caution when chasing down tk warnings about trying to do things after app destroyed. Re-reading the code, deleting it or not seems like it should have no effect and deleting it now has none that I can see on Windows. If either of you test deletion and get no new warnings on Mac, I will patch the deletion away.

Macosx assumes that it is run once on IDLE startup. It is not intended to be idempotent. We could catch the IndexError and ignore it, but that could mask a real error arising from modifying the initial menu structure. We could catch the error and add a note to the message: "Ignore this if this only appears when regrtest re-runs test_idle after a failure." Another approach would be to break the triple indexing into three statements, so we would know which indexing fails. (I suspect the first.)

query and test_query are 3.6 only. test_textview is all 3 versions. I presume the macosx issue is also.

[terryjreedy]

Mark, the 'question above' was actually posted to bpo-27380, the original query issue.

[terryjreedy]

Mark, the 'question above' was actually posted to bpo-27380, the original query issue.

[terryjreedy]

For test_textview, I triggered the symptom -- 4 'TV' NameErrors on re-run, by (temporarily) adding 1/0 to tearDownModule and running with -w. After removing the TV deletion, only the ZeroDivisionError remained.

This experiment also resulted in a '''can't invoke "event" command: application has been destroyed''' message. For me it only happens on the re-run, not in the first run. In verbose mode, it follows the one test in test_config_key. Converting the test to 'pass' proved it to be the source. Looking at the code, I could not find any obvious callback that would be left behind. Adding 'root.update() stops the message. (Adding root.update_idletasks() did not.)

In the output quoted above, the same message is shown for the non-verbose initial run, If from the same cause, the same fix might work on Mac also.

[python-dev]

New changeset b10a312f6d00 by Terry Jan Reedy in branch 'default':
Issue bpo-27714: text_textview now passes when re-run in the same process
https://hg.python.org/cpython/rev/b10a312f6d00

[ned-deily]

Terry, b10a312f6d00 appears to fix most of the issue, that is, if I insert a deliberate failure into another IDLE test and use -w without -j, the four repeatable errors I identified above no longer occur. Thanks. However, the fifth, timing-dependent error still occurs during the -w rerun perhaps ever 5th run or so. It might be worth pursuing to help squash other intermittent results:

======================================================================
ERROR: setUpClass (idlelib.idle_test.test_autocomplete.AutoCompleteTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
    macosx.setupApp(cls.root, None)
  File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
    overrideRootMenu(root, flist)
  File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
    del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of range

[python-dev]

New changeset dc31fa499359 by Terry Jan Reedy in branch '3.5':
Issue bpo-27714: text_textview now passes when re-run in the same process
https://hg.python.org/cpython/rev/dc31fa499359

[terryjreedy]

I agree. The issue may come up with other tests in the future. I just decided to do the part I knew how to do first ;-).

For bpo-18409, I added the mac call to Phil's original patch with the vague comment "now needed to run without raising". But it is not needed on Windows. I suspect that the then new IDLE GSOC student, S.H., had emailed me after reviewing existing work. He mostly worked on Linux, but I believe he had access to a Mac.

setupApp calls 4 fixup functions. I am pretty sure that that the test itself does not need the failing menu fix overrideRootMenu, at least not now. (The case might be different if I ever add live EditorWindow tests.) Three days after the initial patch, the patch by S.H. for bpo-21682 replaced EditorWindow with a mock. The reason was to stop menu leaks. But it should have also eliminated any need to patch the EditorWindow menu.

Two of the functions, addOpenEventSupport and hideTkConsole don't seem like they should ever be needed. The last, should only matter for tests that generate the events involved (and only on 8.5).

So what happens if you comment out line 34?
macosx.setupApp(cls.root, None)

If deleting the call is not possible, maybe we can replace it with something more specific. If not, then the following should work, but at the cost of masking a possible IDLE bug (unless a new _utest=False parameter is added to the signature -- but I dislike adding such complications until really needed.)

first_time = True  # new
def setupApp(root, flist):
    ""
    if isAquaTk() and first_time:  # modified
        first_time = False  # new
        ...

[python-dev]

New changeset 0de509a79181 by Terry Jan Reedy in branch '2.7':
Issue bpo-27714: For IDLE's test_textview, backport 3.x subclass with mocks
https://hg.python.org/cpython/rev/0de509a79181

[terryjreedy]

The issue with test_textview in 3.x is that tearDownModule deleted a module attribute, 'TV', that was not created in setUpModule. For 2.7, 'TV' is a class attribute that is created in setUpClass and deleted in tearDownClass, so it does not have the same problem. On the other hand, 'TV' is the original textView.TextViewer class, as modified with mocks in setUpClass. The modifications are not reverted in tearDownClass. This is the original code, later modified in 3.x. This is safe both for re-runs and likely future tests, but this is not immediately obvious. In any case, it violates the idea/rule that tests should not make permanent changes. So I backported the local subclass of TextViewer used in 3.x.

[ned-deily]

So what happens if you comment out line 34?
macosx.setupApp(cls.root, None)

That seems to eliminate the intermittent "IndexError: list assignment index out of range" and doesn't seem to cause any new errors.