test_tools leaks randomly references on x86 Gentoo Refleaks 3.6 and 3.x

[vstinner]

BPO	31174
Nosy	@vstinner, @serhiy-storchaka
PRs	bpo-31227: regrtest reseeds RNG before each test #3059 bpo-31174: Fix test_tools.test_unparse #4102 [3.6] bpo-31174: Fix test_tools.test_unparse (GH-4102) #4104 bpo-31174: Improve the code of test_tools.test_unparse. #4146 [3.6] bpo-31174: Improve the code of test_tools.test_unparse. (GH-4146) #4148

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-24.11:18:27.089>
created_at = <Date 2017-08-10.11:45:39.097>
labels = ['3.7', 'tests']
title = 'test_tools leaks randomly references on x86 Gentoo Refleaks 3.6 and 3.x'
updated_at = <Date 2017-10-27.13:17:38.975>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-10-27.13:17:38.975>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-24.11:18:27.089>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-08-10.11:45:39.097>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31174
keywords = ['patch']
message_count = 11.0
messages = ['300075', '300439', '302063', '304889', '304890', '304897', '304898', '304900', '304901', '305117', '305119']
nosy_count = 2.0
nosy_names = ['vstinner', 'serhiy.storchaka']
pr_nums = ['3059', '4102', '4104', '4146', '4148']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31174'
versions = ['Python 3.6', 'Python 3.7']

[vstinner]

The "x86 Gentoo Refleaks 3.x" buildbot runs tests using -u-cpu to disable the cpu resource. The problem is that DirectoryTestCase.test_files() of Lib/test/test_tools/test_unparse.py uses random:

        # Test limited subset of files unless the 'cpu' resource is specified.
        if not test.support.is_resource_enabled("cpu"):
            names = random.sample(names, 10)

So when we run the same test 7 times, each run uses different data.

I see different options:

• Reseed random using the same seed in dash_R() of regrtest
• Always test all data in test_unparse.py: all files, or select a specific set of interesting files

The random issue is more generic than just test_unparse.py, and so it would be interesting to explore this path. Maybe the random issue explains why some other tests fail randomly.

libregrtest always seeds the random RNG using a seed displayed on the standard output. We should either reuse this seed, or create a new unique seed for each test file, and display it (to be able to reproduce tests).

Reseed random before running each test file can also helps to make tests more reproductible

--

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/52/steps/test/logs/stdio

test_tools leaked [1, 4, 2] memory blocks, sum=7
(...)
Re-running test 'test_tools' in verbose mode
(...)
test_tools leaked [1, 2, 2] memory blocks, sum=5
(...)
1 test failed again:
test_tools

[vstinner]

I created bpo-31227: "regrtest: reseed random with the same seed before running a test file".

[vstinner]

test_tools also fails on x86 Gentoo Refleaks 3.6:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.6/builds/90

test_tools leaked [1, 1, 118] memory blocks, sum=120
(...)
test_tools leaked [7, 1, 6] memory blocks, sum=14

[vstinner]

"test_tools leaked (...)" still occurs randomly on 3.6 and 3.x Gentoo buildbot.

[vstinner]

My previous attempt, PR 3059, was a a very generic fix, but my bpo-31227 idea was rejected.

I proposed a new idea, PR 4102, which is restricted to this specific issue:
"test_unparse.DirectoryTestCase now stores the names sample to always
test the same files. It prevents false alarms when hunting reference
leaks."

[vstinner]

New changeset 8e482be by Victor Stinner in branch 'master':
bpo-31174: Fix test_tools.test_unparse (bpo-4102)
8e482be

[vstinner]

Python 2.7 is not affected. Even if Python 2.7 has Demo/parser/test_unparse.py, this test is not part of the Python test suite, and it doesn't use random.sample(names, 10).

[vstinner]

New changeset d8f78a1 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
bpo-31174: Fix test_tools.test_unparse (GH-4102) (bpo-4104)
d8f78a1

[vstinner]

The bug should now be fixed.

[serhiy-storchaka]

New changeset 7351f9e by Serhiy Storchaka in branch 'master':
bpo-31174: Improve the code of test_tools.test_unparse. (bpo-4146)
7351f9e

[serhiy-storchaka]

New changeset bb78898 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-31174: Improve the code of test_tools.test_unparse. (GH-4146) (bpo-4148)
bb78898