CI job for parallel frontend ui tests#158307
Conversation
It's added to give threads inside the compiler opportunity to use multiple cores simultaneously, which gives better chance for bugs to occur. |
| tests/codegen-llvm \ | ||
| tests/codegen-units \ | ||
| -- \ | ||
| --parallel-frontend-threads=4 --iteration-count=2 \ |
There was a problem hiding this comment.
Only tests/ui tests need to be run in this mode.
For non-TestMode::Ui suites --parallel-frontend-threads=4 doesn't do anything anyway, and running ui-fulldeps is just not important enough.
The second run can then --exclude tests/ui.
| tests/codegen-llvm \ | ||
| tests/codegen-units \ | ||
| -- \ | ||
| --parallel-frontend-threads=4 --iteration-count=2 \ |
There was a problem hiding this comment.
The N in --iteration-count=N depends on the available CI resources.
The more the better, of course, but right now even one iteration would be acceptable, I think (see "Enabling parallel UI test suite on CI" in rust-lang/compiler-team#1005).
Note that RUST_TEST_THREADS=1 makes things really slow.
I'll leave this for the infra team to decide.
There was a problem hiding this comment.
I will rerun the job again to check the time. Generally speaking, I would expect that this test job does not become the slowest job which is generally around 3h 15m.
|
r? @jieyouxu |
Oh, indeed.
I believe it can all be tweaked with a script though? For instance, we could use something like Which would improve performance drastically. This introduces machine-dependency, but maybe it's not big of an issue here |
|
@petrochenkov could you grant me bors capabilities, please? |
|
@bors delegate=try |
|
✌️ @heinwol, you can now perform try builds on this pull request! Warning You used the legacy format of the delegate command. You can now post |
|
@bors try jobs=optional-x86_64-gnu-parallel-frontend |
This comment has been minimized.
This comment has been minimized.
Attempt to run parallel frontend CI try-job: optional-x86_64-gnu-parallel-frontend
This comment has been minimized.
This comment has been minimized.
* attempt to run parallel CI * fix `tests/rustdoc-ui/doctest/no-capture.rs` fail by setting `RUST_TEST_THREADS=1` for ui tests only * more efficient implementation There are two limitations though for the thread count to work properly: 1. The machine is not performing any other tasks 2. It has only efficiency cores Anyway, it's up to the infra team to decide if this design is ok * small adjustments * fix: make sure `RUST_TEST_THREADS >= 1`
|
🔨 5 commits were squashed into 8118399. |
3ed3e7a to
8118399
Compare
This comment was marked as resolved.
This comment was marked as resolved.
| --enable-sanitizers \ | ||
| --enable-profiler \ | ||
| --enable-compiler-docs \ |
There was a problem hiding this comment.
NB: oh, you could probably save some time (e.g. by not having to build rustdocs for this), though this doesn't really matter much compared to impact of RUST_TEST_THREADS I'd expect. So I'd just leave this in.
…eyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes rust-lang#157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
…eyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes rust-lang#157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
…eyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes rust-lang#157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
…eyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes rust-lang#157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
…uwer Rollup of 25 pull requests Successful merges: - #158871 (add relnotes for 1.97.0) - #158968 (stdarch subtree update) - #154445 (rustdoc: Represent `--output-format=json` coverage and ir differently) - #156370 (Reject linked dylib EII default overrides) - #157153 (allow `Allocator`s to be used as `#[global_allocator]`s) - #158495 (Rename HAS_CT_PROJECTION to HAS_CONST_ALIAS) - #158617 (allow mGCA const arguments to fall back to anon consts) - #158645 (Fix splat ICEs and ban it in closures) - #158655 (Fix coroutine MIR saved local remapping) - #158666 (Carry the `b_offset` inside `BackendRepr::ScalarPair`) - #158912 (Introduce new bootstrap config section for PGO configuration) - #158920 (Update wasm-component-ld to 0.5.26) - #158926 (wrapping_sh* methods: clarify underspecified reference) - #158927 (add core test run with `-Zforce-intrinsic-fallback`) - #158932 (Do not build the compiler when invoking `x perf compare`) - #158937 (Emit the emscripten entry point as `__main_argc_argv`) - #151379 (Stabilize `VecDeque::retain_back` from `truncate_front`) - #156548 ( Library support for aarch64-unknown-linux-pauthtest target) - #158307 (CI job for parallel frontend ui tests) - #158347 (Improve generic parameters handling for #[diagnostic::on_const]) - #158722 (delegation: do not always inherit `ConstArgHasType` predicates) - #158741 (Simplify `Option::into_flat_iter` signature) - #158807 (Add regression test for CString::clone_into unwind safety) - #158862 (Fix the span for parameter suggestion ) - #158883 (tests: fix enum-match.rs to handle LLVM 23)
…eyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes rust-lang#157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
Rollup of 25 pull requests Successful merges: - #158871 (add relnotes for 1.97.0) - #158968 (stdarch subtree update) - #157690 (codegen_ssa: pack small const aggregates into immediate stores) - #158541 (Move `std::io::Write` to `core::io`) - #154445 (rustdoc: Represent `--output-format=json` coverage and ir differently) - #156370 (Reject linked dylib EII default overrides) - #157153 (allow `Allocator`s to be used as `#[global_allocator]`s) - #158495 (Rename HAS_CT_PROJECTION to HAS_CONST_ALIAS) - #158617 (allow mGCA const arguments to fall back to anon consts) - #158645 (Fix splat ICEs and ban it in closures) - #158655 (Fix coroutine MIR saved local remapping) - #158666 (Carry the `b_offset` inside `BackendRepr::ScalarPair`) - #158870 (std: merge the unix-like io::error modules into one file) - #158920 (Update wasm-component-ld to 0.5.26) - #158926 (wrapping_sh* methods: clarify underspecified reference) - #158927 (add core test run with `-Zforce-intrinsic-fallback`) - #158932 (Do not build the compiler when invoking `x perf compare`) - #158937 (Emit the emscripten entry point as `__main_argc_argv`) - #151379 (Stabilize `VecDeque::retain_back` from `truncate_front`) - #156144 (Better docs for PartialEq (includes macro rename)) - #156548 ( Library support for aarch64-unknown-linux-pauthtest target) - #158307 (CI job for parallel frontend ui tests) - #158347 (Improve generic parameters handling for #[diagnostic::on_const]) - #158722 (delegation: do not always inherit `ConstArgHasType` predicates) - #158741 (Simplify `Option::into_flat_iter` signature)
…eyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes rust-lang#157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
Rollup of 23 pull requests Successful merges: - #158968 (stdarch subtree update) - #154445 (rustdoc: Represent `--output-format=json` coverage and ir differently) - #158495 (Rename HAS_CT_PROJECTION to HAS_CONST_ALIAS) - #158666 (Carry the `b_offset` inside `BackendRepr::ScalarPair`) - #158870 (std: merge the unix-like io::error modules into one file) - #158920 (Update wasm-component-ld to 0.5.26) - #158926 (wrapping_sh* methods: clarify underspecified reference) - #158927 (add core test run with `-Zforce-intrinsic-fallback`) - #158932 (Do not build the compiler when invoking `x perf compare`) - #158937 (Emit the emscripten entry point as `__main_argc_argv`) - #151379 (Stabilize `VecDeque::retain_back` from `truncate_front`) - #156144 (Better docs for PartialEq (includes macro rename)) - #156548 ( Library support for aarch64-unknown-linux-pauthtest target) - #157995 (`Vec::dedup_by` docs explicit function argument order) - #158307 (CI job for parallel frontend ui tests) - #158741 (Simplify `Option::into_flat_iter` signature) - #158807 (Add regression test for CString::clone_into unwind safety) - #158862 (Fix the span for parameter suggestion ) - #158894 (Make the ordering of non-terminal binds in ambiguity error messages deterministic) - #158902 (add codegen test for range length bound propagation) - #158913 (Update `browser-ui-test` version to `0.24.1`) - #158935 (std: support real fd methods on Emscripten) - #158978 (Add regression test for too-big by-value ABI args)
Rollup merge of #158307 - heinwol:parallel-frontend-CI, r=jieyouxu CI job for parallel frontend ui tests ## Summary Part of rust-lang/compiler-team#1005. Supersedes #157705. ### Initial setup in this PR For the initial setup in this PR, we'll go ahead with the following combination: - `RUST_TEST_THREADS`: `max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})` - `--parallel-frontend-threads`: **4** - `--iteration-count`: **2** Against `./x test tests/ui --stage=2` only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m). ## Additional context ### Issues with the previous attempt The issue with the original change was arguments `--parallel-frontend-threads=4 --iteration-count=2` being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors. I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like `x test tests/` it runs _all_ the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. `--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...` is still not exhaustive list of exclusions. I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in [bootstrap::core::builder::Builder](https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/struct.Builder.html#structfield.log_cli_step_for_tests) for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like. r? @petrochenkov --- try-job: optional-x86_64-gnu-parallel-frontend
Rollup of 23 pull requests Successful merges: - rust-lang/rust#158968 (stdarch subtree update) - rust-lang/rust#154445 (rustdoc: Represent `--output-format=json` coverage and ir differently) - rust-lang/rust#158495 (Rename HAS_CT_PROJECTION to HAS_CONST_ALIAS) - rust-lang/rust#158666 (Carry the `b_offset` inside `BackendRepr::ScalarPair`) - rust-lang/rust#158870 (std: merge the unix-like io::error modules into one file) - rust-lang/rust#158920 (Update wasm-component-ld to 0.5.26) - rust-lang/rust#158926 (wrapping_sh* methods: clarify underspecified reference) - rust-lang/rust#158927 (add core test run with `-Zforce-intrinsic-fallback`) - rust-lang/rust#158932 (Do not build the compiler when invoking `x perf compare`) - rust-lang/rust#158937 (Emit the emscripten entry point as `__main_argc_argv`) - rust-lang/rust#151379 (Stabilize `VecDeque::retain_back` from `truncate_front`) - rust-lang/rust#156144 (Better docs for PartialEq (includes macro rename)) - rust-lang/rust#156548 ( Library support for aarch64-unknown-linux-pauthtest target) - rust-lang/rust#157995 (`Vec::dedup_by` docs explicit function argument order) - rust-lang/rust#158307 (CI job for parallel frontend ui tests) - rust-lang/rust#158741 (Simplify `Option::into_flat_iter` signature) - rust-lang/rust#158807 (Add regression test for CString::clone_into unwind safety) - rust-lang/rust#158862 (Fix the span for parameter suggestion ) - rust-lang/rust#158894 (Make the ordering of non-terminal binds in ambiguity error messages deterministic) - rust-lang/rust#158902 (add codegen test for range length bound propagation) - rust-lang/rust#158913 (Update `browser-ui-test` version to `0.24.1`) - rust-lang/rust#158935 (std: support real fd methods on Emscripten) - rust-lang/rust#158978 (Add regression test for too-big by-value ABI args)
View all comments
Summary
Part of rust-lang/compiler-team#1005.
Supersedes #157705.
Initial setup in this PR
For the initial setup in this PR, we'll go ahead with the following combination:
RUST_TEST_THREADS:max(1, $(nproc) // ${PARALLEL_FRONTEND_THREADS})--parallel-frontend-threads: 4--iteration-count: 2Against
./x test tests/ui --stage=2only. We can tune these knobs in follow-ups. In try jobs we ran, this should not exceed the current longest auto job duration (at around 3h 15m).Additional context
Issues with the previous attempt
The issue with the original change was arguments
--parallel-frontend-threads=4 --iteration-count=2being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors.I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like
x test tests/it runs all the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests.--skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ...is still not exhaustive list of exclusions.I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in bootstrap::core::builder::Builder for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like.
r? @petrochenkov
try-job: optional-x86_64-gnu-parallel-frontend