multiple tests in test_tools fail since Python 3.9

[felixonmars]

BPO	41971
Nosy	@ericsnowcurrently, @felixonmars, @yan12125, @corona10, @pablogsal, @tirkarthi
PRs	bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state #22652 [3.9] bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) #22653

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-11.20:35:33.917>
created_at = <Date 2020-10-07.23:07:56.299>
labels = ['tests', '3.9']
title = 'multiple tests in test_tools fail since Python 3.9'
updated_at = <Date 2020-10-11.20:54:42.966>
user = 'https://github.com/felixonmars'

bugs.python.org fields:

activity = <Date 2020-10-11.20:54:42.966>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-11.20:35:33.917>
closer = 'pablogsal'
components = ['Tests']
creation = <Date 2020-10-07.23:07:56.299>
creator = 'felixonmars'
dependencies = []
files = []
hgrepos = []
issue_num = 41971
keywords = ['patch']
message_count = 7.0
messages = ['378191', '378194', '378438', '378446', '378450', '378453', '378454']
nosy_count = 6.0
nosy_names = ['eric.snow', 'felixonmars', 'yan12125', 'corona10', 'pablogsal', 'xtreak']
pr_nums = ['22652', '22653']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41971'
versions = ['Python 3.9']

[felixonmars]

I am packaging Python for Arch and the test suite of Python 3.8.6 passes here without these:

======================================================================
FAIL: test_multiple_roots (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 118, in test_multiple_roots
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_w[26 chars]tion walk at 0x7ff2eb562f70>)), ('_walk', ('eg[42 chars]0>))] != [('_w[26 chars]tion _walk_tree at 0x7ff2e5babdc0>)), ('_walk'[54 chars]0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>)),
  ? ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>)),
? + +++++ + ^^^^

• ('_walk', ('eggs', '.c', <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ ('_walk', ('eggs', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_multiple_suffixes (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 189, in test_multiple_suffixes
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_no_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests) [None]
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 209, in test_no_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_no_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests) []
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 209, in test_no_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '', <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', '', <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', '', <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_no_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests) [()]
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 209, in test_no_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', (), <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', (), <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', (), <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', (), <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', (), <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', (), <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_one_root (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 99, in test_one_root
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_one_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 167, in test_one_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_relparent (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 228, in test_relparent
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_w[33 chars]tion walk at 0x7ff2eb562f70>)), ('_walk', ('/x[49 chars]0>))] != [('_w[33 chars]tion _walk_tree at 0x7ff2e5babdc0>)), ('_walk'[61 chars]0>))]

First differing element 0:
('_walk', ('/x/y/z/spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('/x/y/z/spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('/x/y/z/spam', '.c', <function walk at 0x7ff2eb562f70>)),
  ? ^^^^^

+ [('_walk', ('/x/y/z/spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>)),
? + +++++ + ^^^^

• ('_walk', ('/x/y/z/eggs', '.c', <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ ('_walk', ('/x/y/z/eggs', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

test test_tools failed
======================================================================
FAIL: test_single_root (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 82, in test_single_root
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_single_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 146, in test_single_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

======================================================================
FAIL: test_typical (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 64, in test_typical
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_w[26 chars]tion walk at 0x7ff2eb562f70>)), ('_walk', ('eg[42 chars]0>))] != [('_w[26 chars]tion _walk_tree at 0x7ff2e5babdc0>)), ('_walk'[54 chars]0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

• [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>)),
  ? ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>)),
? + +++++ + ^^^^

• ('_walk', ('eggs', None, <function walk at 0x7ff2eb562f70>))]
  ? ^^^^^

+ ('_walk', ('eggs', None, <function _walk_tree at 0x7ff2e5babdc0>))]
? + +++++ + ^^^^

----------------------------------------------------------------------

Ran 184 tests in 3.357s

FAILED (failures=11, skipped=2, expected failures=14)
0:25:39 load avg: 0.95 [362/424/3] test_trace -- test_tools failed

[pablogsal]

All our buildbots are running these test without any problem and I cannot reproduce this locally after a fresh install from the binaries in 3.9, so my guess is that there must be something going on on your end.

❯ ./installed_dir/bin/python3 -m test test_tools
0:00:00 load avg: 6.90 Run tests sequentially
0:00:00 load avg: 6.90 [1/1] test_tools

== Tests result: SUCCESS ==

1 test OK.

Total duration: 40 ms
Tests result: SUCCESS

~/Downloads/Python-3.9.0
❯ ./python -m test test_tools
0:00:00 load avg: 6.43 Run tests sequentially
0:00:00 load avg: 6.43 [1/1] test_tools

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.5 sec
Tests result: SUCCESS

[yan12125]

Looks like those failures in test_tools happen only when test_check_c_globals is imported and a test in test_imp is run.

$ cat matchfile
test.test_tools.*
test.test_imp.ImportTests.test_load_from_source

$ ./python -m test --matchfile matchfile test_check_c_globals test_imp test_tools
0:00:00 load avg: 0.90 Run tests sequentially
0:00:00 load avg: 0.90 [1/3] test_check_c_globals
0:00:00 load avg: 0.90 [2/3] test_imp -- test_check_c_globals run no tests
0:00:00 load avg: 0.90 [3/3] test_tools
test test_tools failed -- multiple errors occurred; run in verbose mode for details
test_tools failed

== Tests result: FAILURE ==

1 test OK.

1 test failed:
    test_tools

1 test run no tests:
    test_check_c_globals

Total duration: 1.9 sec
Tests result: FAILURE

If I remove test_check_c_globals from the second command or test.test_imp.ImportTests.test_load_from_source from matchfile, test_tools does not fail.

[pablogsal]

Adding Eric Snow as this seems to be something going on on the c_analyzer

[pablogsal]

Seems that the problem is that this check:

https://github.com/python/cpython/blob/master/Tools/c-analyzer/c_analyzer/common/files.py#L90

is done using a default value for "get_files":

https://github.com/python/cpython/blob/master/Tools/c-analyzer/c_analyzer/common/files.py#L63

But the import from source creates a new function for os.walk and therefore the comparison fails.

[pablogsal]

New changeset 47ecfd8 by Pablo Galindo in branch 'master':
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652)
47ecfd8

[pablogsal]

New changeset 4af6729 by Miss Skeleton (bot) in branch '3.9':
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) (GH-22653)
4af6729