https://buildbot.python.org/all/#/builders/597/builds/178

test_error (idlelib.idle_test.test_run.HandleErrorTest) ... FAIL
(...)

======================================================================
FAIL: test_error (idlelib.idle_test.test_run.HandleErrorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.8.cstratak-RHEL8-aarch64.refleak/build/Lib/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/home/buildbot/buildarea/3.8.cstratak-RHEL8-aarch64.refleak/build/Lib/idlelib/idle_test/test_run.py", line 352, in test_error
    eq(self.func.called, 2)
AssertionError: 4 != 2



I can reproduce the issue on 3.8, 3.9 and master branches:

14:00:18 vstinner@apu$ ./python -m test -m test_error -v test_idle  test_idle  
== CPython 3.8.5+ (heads/3.8:61f23cb62d, Aug 10 2020, 14:00:04) [GCC 10.2.1 20200723 (Red Hat 10.2.1-1)]
== Linux-5.7.11-200.fc32.x86_64-x86_64-with-glibc2.29 little-endian
== cwd: /home/vstinner/python/3.8/build/test_python_43858
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.68 Run tests sequentially
0:00:00 load avg: 1.68 [1/2] test_idle
test_error (idlelib.idle_test.test_run.HandleErrorTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.004s

OK
0:00:00 load avg: 1.68 [2/2] test_idle
test_error (idlelib.idle_test.test_run.HandleErrorTest) ... FAIL

======================================================================
FAIL: test_error (idlelib.idle_test.test_run.HandleErrorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/3.8/Lib/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/home/vstinner/python/3.8/Lib/idlelib/idle_test/test_run.py", line 352, in test_error
    eq(self.func.called, 2)
AssertionError: 4 != 2

----------------------------------------------------------------------

Ran 1 test in 0.007s

FAILED (failures=1)
test test_idle failed
test_idle failed

== Tests result: FAILURE ==

1 test OK.

1 test failed:
    test_idle

Total duration: 722 ms
Tests result: FAILURE


It seems like the test was added recently:

commit f2e161c27964a59bc5ab20d96f87ba5862c6222d
Author: Terry Jan Reedy <[email protected]>
Date:   Sun Aug 9 16:08:30 2020 -0400

    bpo-41468: Improve and test IDLE run error exit  (GH-21798)
    
    A message box pops up when an unexpected error stops the run process.  Tell users it is likely a random glitch, but report it if not.

Your reproducer worked to reproduce the bug for me, make the cause obvious, and test the fix.

Łukasz, it would be good if this test fix made it into the release candidate, but probably not essential as not too many people will check it for refleaks.

New changeset 416f0b71ba84fe83ee2ba4399b8a28712702980b by Terry Jan Reedy in branch 'master':
bpo-41514: Fix buggy IDLE test (GH-21808)
https://github.com/python/cpython/commit/416f0b71ba84fe83ee2ba4399b8a28712702980b

issue 41514 added the new test.

New changeset 860bc0ea70c365825bfd9b7de7685cf6842ca3c7 by Miss Islington (bot) in branch '3.8':
bpo-41514: Fix buggy IDLE test (GH-21808)
https://github.com/python/cpython/commit/860bc0ea70c365825bfd9b7de7685cf6842ca3c7

New changeset 9c253f4bc9dbf711dce3fffeaef86179c29fa0f5 by Miss Islington (bot) in branch '3.9':
bpo-41514: Fix buggy IDLE test (GH-21808)
https://github.com/python/cpython/commit/9c253f4bc9dbf711dce3fffeaef86179c29fa0f5

I just read that the release is tomorrow, so no issue about including this.