Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest

[vstinner]

BPO	26295
Nosy	@brettcannon, @vstinner, @vadmium, @serhiy-storchaka
Files	test_regrtest_tmpdir.patch test_regrtest_tmpdir-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-01.19:48:50.127>
created_at = <Date 2016-02-05.09:15:07.194>
labels = ['tests']
title = 'Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest'
updated_at = <Date 2016-04-01.19:48:50.126>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2016-04-01.19:48:50.126>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-04-01.19:48:50.127>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2016-02-05.09:15:07.194>
creator = 'vstinner'
dependencies = []
files = ['42131', '42177']
hgrepos = []
issue_num = 26295
keywords = ['patch']
message_count = 12.0
messages = ['259647', '259649', '259650', '260607', '260609', '261563', '261833', '262430', '262640', '262650', '262651', '262763']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'vstinner', 'python-dev', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26295'
versions = ['Python 3.5', 'Python 3.6']

[vstinner]

test_regrtest creates temporary test files called test_regrtest_pid_xxx.py in Lib/test/. The problem is that some tests like test___all__ and test_zipfile haves test relying on the list of Lib/test/test_*.py.

When tests are run in parallel, test_regrtest can creates temporary test_regrtest_pid_xxx.py files, test_zipfile sees them, test_regrtest removes them, and then test_zipfiles fails.

The best would be to write these temporary files into a temporary directory, but I failed to fix Lib/test/regrtest.py to load tests from a different directory. In theory, Python 3 supports packages with files splitted into multiple directories, in practice it doesn't seem to work :-p

Maybe test_regrtest should use a test submodule like Lib/test/temp_regrtest/ ?

test_regrtest started to create temporary test_xxx.py files since issue bpo-25220. (Other changes to test_regrtest: issues bpo-18174, bpo-22806, bpo-25260, bpo-25306, bpo-25369, bpo-25373, bpo-25694).

======================================================================
ERROR: test_all (test.test___all__.AllTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test___all__.py", line 102, in test_all
    with open(path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/haypo/prog/python/default/Lib/test/test_regrtest_25743_noop20.py'

---

[serhiy-storchaka]

The directory with test files can be read-only. test_regrtest should use temporary directory.

[vstinner]

The directory with test files can be read-only.

test_regrtest skips tests which requires to write in Lib/test/ if the write fails.

        except PermissionError as exc:
            if not sysconfig.is_python_build():
                self.skipTest("cannot write %s: %s" % (path, exc))

test_regrtest should use temporary directory.

Yes! That's the purpose of this issue!

But I was unable to find a technical solution to implement this. I need to try harder :-)

[vadmium]

You say you couldn’t get the test package to be split into multiple directories. What did you try? As far as I can see, we need a way for test_regrtest to invoke the child process with some flag to say “add this extra directory to test.__path__”.

[vadmium]

Another idea is to make a loader script that loads the “test” module, patches its __path__, and then runs test.__main__ or whatever as usual.

[vstinner]

Oh ok, I found a bug in the setup_tests() function of libregrtest: issue bpo-26538.

Attached patch fixes this issue, it requires the patch of the issue issue bpo-26538.

[vstinner]

Hum, test_doctest, test_zipfile and test_pyclbr fail if test is converted to a package (if Lib/test/init.py is removed).

Attached patch fixes test_zipfile.

I created the issue bpo-26569 for test_pyclbr failure.

[vstinner]

I created the issue bpo-26641 for doctest (test_doctest).

[python-dev]

New changeset f7ddd72b70ea by Victor Stinner in branch 'default':
Issue bpo-26295: When using "python3 -m test --testdir=TESTDIR", regrtest doesn't
https://hg.python.org/cpython/rev/f7ddd72b70ea

New changeset c83349c129c9 by Victor Stinner in branch 'default':
Issue bpo-26295: test_regrtest now uses a temporary directory
https://hg.python.org/cpython/rev/c83349c129c9

[python-dev]

New changeset e29f12ed4444 by Victor Stinner in branch 'default':
Issue bpo-26295: Fix test_regrtest.test_tools_buildbot_test()
https://hg.python.org/cpython/rev/e29f12ed4444

[python-dev]

New changeset e47e00723e5d by Victor Stinner in branch 'default':
Issue bpo-26295: Enhanc test_regrtest.test_tools_script_run_tests()
https://hg.python.org/cpython/rev/e47e00723e5d

[vstinner]

Finally, I fixed the issue differently: test_regrtest now creates a temporary directory and uses --testdir command line parameter.

Before, using --testdir didn't work because regrtest always added "test." prefix to module names (ex: test_os => test.test_os). It isn't the case anymore when --testdir is used.

Buildbots are green, so I close the issue.