RFC: issue a warning in regrtest when no tests have been executed?

[vstinner]

BPO	34279
Nosy	@terryjreedy, @vstinner, @serhiy-storchaka, @pablogsal, @miss-islington
PRs	bpo-34279: Issue a warning if no tests have been executed #10150 [3.7] bpo-34279: Synchronize regrtest with master #10800 [2.7] bpo-34279, regrtest: Issue a warning if no tests have been executed #10801 [3.6] [3.7] bpo-34279: Synchronize regrtest with master (GH-10800) #10802 bpo-34279: regrtest consider that skipped tests are ran #11132 [3.7] bpo-34279: regrtest consider that skipped tests are ran (GH-11132) #11155 [3.6] bpo-34279: regrtest consider that skipped tests are ran (GH-11132) #11156 [2.7] bpo-34279: regrtest consider that skipped tests are ran (GH-11132) #11158

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-29.21:58:19.862>
created_at = <Date 2018-07-30.10:48:27.929>
labels = ['3.8', '3.7', 'tests']
title = 'RFC: issue a warning in regrtest when no tests have been executed?'
updated_at = <Date 2018-12-14.12:44:12.403>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-12-14.12:44:12.403>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-29.21:58:19.862>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2018-07-30.10:48:27.929>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34279
keywords = ['patch']
message_count = 12.0
messages = ['322667', '322684', '322783', '323087', '330705', '330720', '330722', '330725', '330728', '331808', '331818', '331824']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'vstinner', 'serhiy.storchaka', 'pablogsal', 'miss-islington']
pr_nums = ['10150', '10800', '10801', '10802', '11132', '11155', '11156', '11158']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34279'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[vstinner]

Python test runner regrtest has many options to select tests: -u (resources), -m and --matchfile, -x (exclude), etc. Sometimes, I do a mistake and I run 0 test, but I still get a success.

Maybe regrtest should be modified to fail with an error when no test is run? At least, it would help to understand my mistake if regrtest could emit a warning in such case.

Example:
---

$ ./python  -W error  -m test test_asyncio -m test_set_default_executor_deprecation_warnings 
Run tests sequentially
0:00:00 load avg: 0.27 [1/1] test_asyncio

== Tests result: SUCCESS ==

1 test OK.

Total duration: 266 ms
Tests result: SUCCESS
---

Success ok, but is there a typo in the test method?

Another example:
---

$ ./python  -W error  -m test test_asyncio -m test_no_such_test -v
== CPython 3.8.0a0 (heads/pr/8533:50c0499950, Jul 30 2018, 12:41:53) [GCC 8.1.1 20180712 (Red Hat 8.1.1-5)]
== Linux-4.17.6-200.fc28.x86_64-x86_64-with-glibc2.26 little-endian
== cwd: /home/vstinner/prog/python/master/build/test_python_5505
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_asyncio

---

Ran 0 tests in 0.001s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 318 ms
Tests result: SUCCESS
---

Success but no test have been executed. It's not obvious at all without the verbose mode:
---

$ ./python  -W error  -m test test_asyncio -m test_no_such_test 
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_asyncio

== Tests result: SUCCESS ==

1 test OK.

Total duration: 260 ms
Tests result: SUCCESS
---

[serhiy-storchaka]

Do we need to output OK and SUCCESS many times? I would prefer shorter and totally different from the normal output in case of no tests ran.

[vstinner]

I would prefer shorter and totally different from the normal output in case of no tests ran.

That's basically the whole purpose of the issue, yep ;-)

[terryjreedy]

A test file that runs no test functions should pass on the buildbots. (If it imports the target file, it is not completely empty. And I can think of a use case for WIP PRs that intentionally do not run anything.)

But I also would like a visually different display, having been fooled more than once. For one thing, one can omit '(unittest.testCase)' from test class definitions ;-).

[vstinner]

New changeset 9724348 by Victor Stinner (Pablo Galindo) in branch 'master':
bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150)
9724348

[vstinner]

New changeset 3600300 by Victor Stinner in branch '2.7':
bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10801)
3600300

[vstinner]

New changeset 8a73cac by Victor Stinner in branch '3.7':
[3.7] bpo-34279: Synchronize regrtest with master (GH-10800)
8a73cac

[miss-islington]

New changeset 43d8126 by Miss Islington (bot) in branch '3.6':
[3.7] bpo-34279: Synchronize regrtest with master (GH-10800)
43d8126

[vstinner]

Thanks Pablo! I merged your change in 2.7, 3.6, 3.7 and master branches.

To reuse the example of my first message, we now get:

vstinner@apu$ ./python -W error -m test test_asyncio -m test_no_such_test
Run tests sequentially
0:00:00 load avg: 1.21 [1/1] test_asyncio
test_asyncio run no tests

== Tests result: NO TEST RUN ==

1 test run no tests:
test_asyncio

Total duration: 347 ms
Tests result: NO TEST RUN

vstinner@apu$ echo $?
0

So the exit status is still a success (code 0), but "Tests result: NO TEST RUN" is an hint that no test has been executed.

If we run two files, but one file runs no test, the output now contains "1 test run no tests: test_sys":

$ ./python -m test test_os test_sys -m test_access -v
== CPython 3.8.0a0 (heads/platform_popen:4fcc412d3d, Nov 29 2018, 21:21:53) [GCC 8.2.1 20181105 (Red Hat 8.2.1-5)]
== Linux-4.19.3-300.fc29.x86_64-x86_64-with-glibc2.28 little-endian
== cwd: /home/vstinner/prog/python/master/build/test_python_18420
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 1.05 [1/2] test_os
test_access (test.test_os.FileTests) ... ok

---

Ran 1 test in 0.001s

OK
0:00:00 load avg: 1.05 [2/2] test_sys

----------------------------------------------------------------------

Ran 0 tests in 0.000s

OK
test_sys run no tests

== Tests result: SUCCESS ==

1 test OK.

1 test run no tests:
test_sys

Total duration: 240 ms
Tests result: SUCCESS

[vstinner]

New changeset 3a8f4fe by Victor Stinner in branch 'master':
bpo-34279: regrtest consider that skipped tests are ran (GH-11132)
3a8f4fe