bpo-38502: regrtest uses process groups if available#16829
Merged
vstinner merged 1 commit intopython:masterfrom Oct 18, 2019
vstinner:regrtest_process_group
Merged
bpo-38502: regrtest uses process groups if available#16829vstinner merged 1 commit intopython:masterfrom vstinner:regrtest_process_group
vstinner merged 1 commit intopython:masterfrom
vstinner:regrtest_process_group
Conversation
Member
Author
|
@nanjekyejoannah @pitrou @pablogsal @gpshead: Do you any drawback in this approach? |
pitrou
reviewed
Oct 17, 2019
Member
pitrou
left a comment
pitrou
left a comment
There was a problem hiding this comment.
Looks good on the principle, but a couple comments.
Lib/test/libregrtest/runtest_mp.py
Outdated
Member
There was a problem hiding this comment.
Shouldn't you call popen.wait() after this?
Member
There was a problem hiding this comment.
fwiw, the old/existing code didn't. (popen.kill() does not call .wait())
Member
Author
There was a problem hiding this comment.
popen.wait() is always called.
The main thread calls stop() which calls _kill().
wait() is called in the TestWorkerProcess thread: _run_process() ends with "finally: self._wait_completed()" to ensure that we also read the exit status to prevent leaking a zombie process.
In my experience, calling wait() from 2 different threads is causing too many troubles :-)
gpshead
reviewed
Oct 17, 2019
Member
gpshead
left a comment
gpshead
left a comment
There was a problem hiding this comment.
Overall I think trying this to see how it pans out. I can't claim to understand all of the specifics of process groups and/or sessions.
Misc/NEWS.d/next/Tests/2019-10-17-00-49-38.bpo-38502.vUEic7.rst
Outdated
Show resolved
Hide resolved
Lib/test/libregrtest/runtest_mp.py
Outdated
Member
There was a problem hiding this comment.
fwiw, the old/existing code didn't. (popen.kill() does not call .wait())
test.regrtest now uses process groups in the multiprocessing mode (-jN command line option) if process groups are available: if os.setsid() and os.killpg() functions are available.
jacobneiltaylor
pushed a commit
to jacobneiltaylor/cpython
that referenced
this pull request
Dec 5, 2019
test.regrtest now uses process groups in the multiprocessing mode (-jN command line option) if process groups are available: if os.setsid() and os.killpg() functions are available.
shihai1991
pushed a commit
to shihai1991/cpython
that referenced
this pull request
Jan 31, 2020
test.regrtest now uses process groups in the multiprocessing mode (-jN command line option) if process groups are available: if os.setsid() and os.killpg() functions are available.
vstinner
added a commit
that referenced
this pull request
Apr 14, 2020
* bpo-37531: regrtest now catchs ProcessLookupError (GH-16827) Fix a warning on a race condition on TestWorkerProcess.kill(): ignore silently ProcessLookupError rather than logging an useless warning. (cherry picked from commit a661392) * bpo-38502: regrtest uses process groups if available (GH-16829) test.regrtest now uses process groups in the multiprocessing mode (-jN command line option) if process groups are available: if os.setsid() and os.killpg() functions are available. (cherry picked from commit ecb035c) * bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989) When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored. For these reasons, add a new option to allow to ignore a list of test and subtests for these situations. (cherry picked from commit e0cd8aa) * regrtest: log timeout at startup (GH-19514) Reduce also worker timeout. (cherry picked from commit 4cf65a6) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
vstinner
added a commit
that referenced
this pull request
Apr 14, 2020
* bpo-36670: regrtest bug fixes (GH-16537) * Fix TestWorkerProcess.__repr__(): start_time is only valid if _popen is not None. * Fix _kill(): don't set _killed to True if _popen is None. * _run_process(): only set _killed to False after calling run_test_in_subprocess(). (cherry picked from commit 2ea71a0) * [3.8] Update libregrtest from master (GH-19516) * bpo-37531: regrtest now catchs ProcessLookupError (GH-16827) Fix a warning on a race condition on TestWorkerProcess.kill(): ignore silently ProcessLookupError rather than logging an useless warning. (cherry picked from commit a661392) * bpo-38502: regrtest uses process groups if available (GH-16829) test.regrtest now uses process groups in the multiprocessing mode (-jN command line option) if process groups are available: if os.setsid() and os.killpg() functions are available. (cherry picked from commit ecb035c) * bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989) When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored. For these reasons, add a new option to allow to ignore a list of test and subtests for these situations. (cherry picked from commit e0cd8aa) * regrtest: log timeout at startup (GH-19514) Reduce also worker timeout. (cherry picked from commit 4cf65a6) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 67b8a1f) * bpo-36842: Fix reference leak in tests by running out-of-proc (GH-13556) (cherry picked from commit 9ddc416) * Backport libregrtest changes from master Co-authored-by: Steve Dower <steve.dower@python.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
regrtest now uses process groups in the multiprocessing mode (-jN
command line option) if process groups are available: if os.setsid()
and os.killpg() functions are available.
https://bugs.python.org/issue38502