Windows: Running the Python test suite sequentially is interrupted by (Pdb) prompt

@gvanrossum

BPO	43955
Nosy	@gvanrossum, @tim-one, @pfmoore, @vstinner, @tjguk, @markshannon, @zware, @serhiy-storchaka, @zooba, @gvanrossum, @corona10, @shihai1991, @shreyanavigyan
PRs	bpo-43955: Handle the case where the distutils warning has already been triggered #25675
Files	tests_log.txt: Full Error Log

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-28.15:27:25.688>
created_at = <Date 2021-04-27.17:03:07.285>
labels = ['3.10', 'OS-windows']
title = 'Windows: Running the Python test suite sequentially is interrupted by (Pdb) prompt'
updated_at = <Date 2021-04-28.15:27:25.687>
user = 'https://github.com/shreyanavigyan'

bugs.python.org fields:

activity = <Date 2021-04-28.15:27:25.687>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-28.15:27:25.688>
closer = 'vstinner'
components = ['Windows']
creation = <Date 2021-04-27.17:03:07.285>
creator = 'shreyanavigyan'
dependencies = []
files = ['49995']
hgrepos = []
issue_num = 43955
keywords = ['patch']
message_count = 29.0
messages = ['392104', '392109', '392114', '392117', '392118', '392119', '392120', '392121', '392132', '392133', '392134', '392135', '392136', '392139', '392140', '392141', '392149', '392150', '392154', '392164', '392182', '392189', '392190', '392194', '392196', '392214', '392216', '392218', '392219']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'tim.peters', 'paul.moore', 'vstinner', 'tim.golden', 'Mark.Shannon', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'Guido.van.Rossum', 'corona10', 'shihai1991', 'shreyanavigyan']
pr_nums = ['25675']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43955'
versions = ['Python 3.10']

Recent test failures have been noticed on Windows 10. The full test log has been attached. The tests test_distutils, test_pdb, test_signal and test_threading require a little bit of attention since there seems to be some kind of error.

I wonder if these four tests do something that affects all threads? The test driver module runs tests in parallel (if you ask it to run all or most tests, like the default).

Pdb error source -

cpython/Lib/test/test_pdb.py

Line 1270 in 9aea31d

def test_pdb_issue_20766():

This is where the error in Pdb is originating from.

The test expects the <built-in default_int_handler> but gets sigint_handler method of a pdb.Pdb instance instead.

Can you submit a PR that fixes it?

I expect parallelism is a red herring: early in the test output attached to this report:

0:00:04 Run tests sequentially

and there's no other evidence in the output that multiple tests are running simultaneously.

Also on Win10, the 4 failing tests here pass for me if I run them one at a time, so it's not obvious.

I _suspect_ that what's going wrong with test_pdb is the root cause: every now & again, for some weeks now, when I try to run tests on Windows I come back to the cmd.exe window and see that it's just sitting there, waiting at a pdb prompt.

In the output attached to this report, note that after test_threading starts,

(Pdb) continue
(Pdb) continue
(Pdb) continue
(Pdb) continue

appears out of the blue. But test_pdb is long finished by that time.

But I'm clueless about current pdb internals.

Yes you're right. Running the failed tests individually does not trigger the errors. And the

(Pdb) continue
(Pdb) continue
(Pdb) continue
(Pdb) continue

actually is manually entered by me. My tests also stop at that prompt but somehow entering these commands again allow us to continue. Why are the errors triggering at all? The tests that are failing when ran alongside the test suite haven't been messed with for a very long time now and still somehow out of the blue they are failing. Any thoughts?

Yes, I also see that pdb prompt. I recommend looking at what different
configurations are used by test.libregrtest.main() when no tests are passed
on the command line.

File "C:\github\cpython\lib\test\test_distutils.py", line 15, in <module>
import distutils.tests
File "C:\github\cpython\lib\contextlib.py", line 140, in __exit__
next(self.gen)
File "C:\github\cpython\lib\test\support\warnings_helper.py", line 179, in _filterwarnings
raise AssertionError("filter (%r, %s) did not catch any warning" %
AssertionError: filter ('The distutils package is deprecated', DeprecationWarning) did not catch any warning

Ah, test_distutils fails if distutils was already imported previously. The test should be fixed.

"""
test test_pdb failed -- Traceback (most recent call last):
  File "C:\github\cpython\lib\doctest.py", line 2205, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_pdb.test_pdb_issue_20766
  File "C:\github\cpython\lib\test\test_pdb.py", line 1270, in test_pdb_issue_20766
"""

This is a surprising failure. No clue why test_pdb fails when other tests are run previously (when tests are run sequentially).

See also bpo-43960

Shreyan Avigyan:

And the "(Pdb) continue (...) actually is manually entered by me.

Do you mean that you modified the Python source code? Does the issue go away if you revert your change or if you test a newly installed Python?

I think the test failures cascade, and some state is not being cleaned up. I'm trying to work through them now, but if someone else is as well, do say so and I'll find something else to do.

Running these 3 tests is enough to reproduce the issue:
-------------------

vstinner@DESKTOP-DK7VBIL C:\vstinner\python\master>python -m test test_interpreters test_pdb test_threading  
Running Debug|x64 interpreter...
0:00:00 Run tests sequentially
0:00:00 [1/3] test_interpreters
0:00:04 [2/3] test_pdb
test test_pdb failed -- Traceback (most recent call last):
  File "C:\vstinner\python\master\lib\doctest.py", line 2205, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_pdb.test_pdb_issue_20766
  File "C:\vstinner\python\master\lib\test\test_pdb.py", line 1270, in test_pdb_issue_20766
(...)
0:00:10 load avg: 0.03 [3/3/1] test_threading -- test_pdb failed
(Pdb)

Moreover, my PR 25674 (bpo-43963) fix these tests.

FYI I used the following patch to debug this issue:

diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
index 872f121127..ee2e1ff77f 100644
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -546,6 +546,8 @@ def _callSetUp(self):
self.setUp()

     def _callTestMethod(self, method):
+        if sys.gettrace() is not None:
+            raise Exception("sys.gettrace() is not None")
         method()

     def _callTearDown(self):

And I used the following script:
---------------

import random
import os.path
import sys

with open("tests") as fp:
    lines = list(filter(bool, (line.strip() for line in fp)))
print(len(lines))

filename = "test"
loop = 2
while os.path.exists(filename):
    filename = "test%s" % loop
    loop += 1

with open(filename, "w") as fp:
    for _ in range(5):
        test = random.choice(lines)
        print(test, file=fp)
    print("test_signal", file=fp)

with open(filename) as fp:
    for line in fp:
        print(line.rstrip())

args = [sys.executable, "-m", "test", "--fromfile=" + filename]
print("RUN", args)

os.execv(args[0], args)

I created the "tests" file using the command:

python -m test --list-tests > tests

Then I removed all tests starting at test_signal, including test_signal. I removed slow tests like asyncio and multiprocessing tests.

Another issue with the tests is that it has become terribly slow. It's taking about 45 minutes to complete 1 run of the test suite. A few days ago, it took only 10 - 15 minutes. And moreover I think test_threading is also modifying sys.gettrace. The logs have the full error message.

--Snippet--

0:33:17 load avg: 0.80 [307/426/2] test_signal
(Pdb) continue
Warning -- sys.gettrace was modified by test_signal
  Before: None
  After:  <bound method Bdb.trace_dispatch of <pdb.Pdb object at 0x0000021A5D2DCD70>>
test test_signal failed -- Traceback (most recent call last):
  File "C:\github\cpython\lib\test\test_signal.py", line 1345, in test_sigint
    signal.raise_signal(signal.SIGINT)
AssertionError: KeyboardInterrupt not raised

--Snippet--

0:38:22 load avg: 0.04 [353/426/3] test_threading
(Pdb) continue
(Pdb) continue
(Pdb) continue
(Pdb) continue
Warning -- sys.gettrace was modified by test_threading
Before: None
After: <bound method Bdb.trace_dispatch of <pdb.Pdb object at 0x0000021A5D2DCD70>>

--Snippet--

Shreyan Avigyan: please try an up to date version of the master branch, I fixed the Pdb issue with commit a09766d.

Same errors.

(Recompilation was done 1 or 2 hours ago against the then updated master branch. It may not include the latest commit)

FYI test_signal is no longer modifying sys.gettrace thanks to your fix. But test_threading still does. Do you have a fix for that?

I ran the test again and test_signal still modifies sys.gettrace()

All the test failures are side effects of other tests because when they are ran individually they do not trigger errors. Running sequentially causes the error.

I ran "python -m test" on an up to date master branch (commit fe52eb6), there is a single remaining issue. test_distutils failed because the deprecation warning was already emited: issue fixed by PR 25675. The other issues are gone.

Shreyan Avigyan: "Same errors."

IMO it's a mistake on your side. You ran an outdated version or you didn't rebuild Python cleanly.

"FYI test_signal is no longer modifying sys.gettrace thanks to your fix. But test_threading still does. (...) I ran the test again and test_signal still modifies sys.gettrace()"

It's not my case on my machine, again it sounds like an issue on your side.

Another issue with the tests is that it has become terribly slow. It's taking about 45 minutes to complete 1 run of the test suite. A few days ago, it took only 10 - 15 minutes.

It took 49 minutes on my Windows 10 VM. Are you sure that you ran the test suite sequentially when it was slower?

I advice you run using -j0 option to run tests in parallel: it's much faster, and it's also more reliable.

== Tests result: FAILURE ==

384 tests OK.

1 test failed:
test_distutils

41 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_smtpnet
test_socketserver test_spwd test_syslog test_threadsignals
test_timeout test_tix test_tk test_ttk_guionly test_urllib2net
test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
test_xxlimited test_xxtestfuzz test_zipfile64

Total duration: 49 min 12 sec
Tests result: FAILURE

New changeset c1a9535 by Steve Dower in branch 'master':
bpo-43955: Handle the case where the distutils warning has already been triggered (GH-25675)
c1a9535

Thanks everyone for fixing these issues and thanks Shreyan for the bug report.

The main initial is now fixed, so I close the issue.

Let's continue the discussions in remaining open issues:

bpo-37387: test_compileall fails randomly on Windows when tests are run in parallel
bpo-43960: test_pdb fails when only some tests are run
bpo-43961 [Windows] test_logging.test_namer_rotator_inheritance() logs a logging error

Navigation Menu

Repository navigation

Windows: Running the Python test suite sequentially is interrupted by (Pdb) prompt #88121

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Windows: Running the Python test suite sequentially is interrupted by (Pdb) prompt #88121

Description

Activity

shreyanavigyan commented on Apr 27, 2021

gvanrossum commented on Apr 27, 2021

shreyanavigyan commented on Apr 27, 2021

shreyanavigyan commented on Apr 27, 2021

gvanrossum commented on Apr 27, 2021

tim-one commented on Apr 27, 2021

shreyanavigyan commented on Apr 27, 2021

gvanrossum commented on Apr 27, 2021

vstinner commented on Apr 27, 2021

vstinner commented on Apr 27, 2021

zooba commented on Apr 27, 2021

8 remaining items

vstinner commented on Apr 27, 2021

vstinner commented on Apr 27, 2021

shreyanavigyan commented on Apr 28, 2021

vstinner commented on Apr 28, 2021

shreyanavigyan commented on Apr 28, 2021

shreyanavigyan commented on Apr 28, 2021

shreyanavigyan commented on Apr 28, 2021

shreyanavigyan commented on Apr 28, 2021

vstinner commented on Apr 28, 2021

vstinner commented on Apr 28, 2021

zooba commented on Apr 28, 2021

vstinner commented on Apr 28, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions