Skip to content

Commit 5ff1eab

Browse files
pmarchinipmarchini
authored
test_runner: support test order randomization
PR-URL: #61747 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
1 parent 25443db commit 5ff1eab

28 files changed

+1557
-21
lines changed

‎doc/api/cli.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2784,6 +2784,38 @@ changes:
27842784
Configures the test runner to only execute top level tests that have the `only`
27852785
option set. This flag is not necessary when test isolation is disabled.
27862786

2787+
### `--test-random-seed`
2788+
2789+
<!-- YAML
2790+
added: REPLACEME
2791+
-->
2792+
2793+
Set the seed used to randomize test execution order. This applies to both test
2794+
file execution order and queued tests within each file. Providing this flag
2795+
enables randomization implicitly, even without `--test-randomize`.
2796+
2797+
The value must be an integer between `0` and `4294967295`.
2798+
2799+
This flag cannot be used with `--watch` or `--test-rerun-failures`.
2800+
2801+
### `--test-randomize`
2802+
2803+
<!-- YAML
2804+
added: REPLACEME
2805+
-->
2806+
2807+
Randomize test execution order. This applies to both test file execution order
2808+
and queued tests within each file. This can help detect tests that rely on
2809+
shared state or execution order.
2810+
2811+
The seed used for randomization is printed in the test summary and can be
2812+
reused with `--test-random-seed`.
2813+
2814+
For detailed behavior and examples, see
2815+
[randomizing tests execution order][].
2816+
2817+
This flag cannot be used with `--watch` or `--test-rerun-failures`.
2818+
27872819
### `--test-reporter`
27882820

27892821
<!-- YAML
@@ -3702,6 +3734,8 @@ one is included in the list below.
37023734
* `--test-isolation`
37033735
* `--test-name-pattern`
37043736
* `--test-only`
3737+
* `--test-random-seed`
3738+
* `--test-randomize`
37053739
* `--test-reporter-destination`
37063740
* `--test-reporter`
37073741
* `--test-rerun-failures`
@@ -4286,6 +4320,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
42864320
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
42874321
[module compile cache]: module.md#module-compile-cache
42884322
[preloading asynchronous module customization hooks]: module.md#registration-of-asynchronous-customization-hooks
4323+
[randomizing tests execution order]: test.md#randomizing-tests-execution-order
42894324
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
42904325
[running tests from the command line]: test.md#running-tests-from-the-command-line
42914326
[scavenge garbage collector]: https://v8.dev/blog/orinoco-parallel-scavenger

‎doc/api/test.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -627,6 +627,94 @@ prevent shell expansion, which can reduce portability across systems.
627627
node --test "**/*.test.js" "**/*.spec.js"
628628
```
629629

630+
### Randomizing tests execution order
631+
632+
<!-- YAML
633+
added: REPLACEME
634+
-->
635+
636+
> Stability: 1.0 - Early development
637+
638+
The test runner can randomize execution order to help detect
639+
order-dependent tests. When enabled, the runner randomizes both discovered
640+
test files and queued tests within each file. Use `--test-randomize` to
641+
enable this mode.
642+
643+
```bash
644+
node --test --test-randomize
645+
```
646+
647+
When randomization is enabled, the test runner prints the seed used for the run
648+
as a diagnostic message:
649+
650+
```text
651+
Randomized test order seed: 12345
652+
```
653+
654+
Use `--test-random-seed=<number>` to replay the same randomized order
655+
deterministically. Supplying `--test-random-seed` also enables randomization,
656+
so `--test-randomize` is optional when a seed is provided:
657+
658+
```bash
659+
node --test --test-random-seed=12345
660+
```
661+
662+
In most test files, randomization works automatically. One important exception
663+
is when subtests are awaited one by one. In that pattern, each subtest starts
664+
only after the previous one finishes, so the runner keeps declaration order
665+
instead of randomizing it.
666+
667+
Example: this runs sequentially and is **not** randomized.
668+
669+
```mjs
670+
import test from 'node:test';
671+
672+
test('math', async (t) => {
673+
for (const name of ['adds', 'subtracts', 'multiplies']) {
674+
// Sequentially awaiting each subtest preserves declaration order.
675+
await t.test(name, async () => {});
676+
}
677+
});
678+
```
679+
680+
```cjs
681+
const test = require('node:test');
682+
683+
test('math', async (t) => {
684+
for (const name of ['adds', 'subtracts', 'multiplies']) {
685+
// Sequentially awaiting each subtest preserves declaration order.
686+
await t.test(name, async () => {});
687+
}
688+
});
689+
```
690+
691+
Using suite-style APIs such as `describe()`/`it()` or `suite()`/`test()`
692+
still allows randomization, because sibling tests are enqueued together.
693+
694+
Example: this remains eligible for randomization.
695+
696+
```mjs
697+
import { describe, it } from 'node:test';
698+
699+
describe('math', () => {
700+
it('adds', () => {});
701+
it('subtracts', () => {});
702+
it('multiplies', () => {});
703+
});
704+
```
705+
706+
```cjs
707+
const { describe, it } = require('node:test');
708+
709+
describe('math', () => {
710+
it('adds', () => {});
711+
it('subtracts', () => {});
712+
it('multiplies', () => {});
713+
});
714+
```
715+
716+
`--test-randomize` and `--test-random-seed` are not supported with `--watch` mode.
717+
630718
Matching files are executed as test files.
631719
More information on the test file execution can be found
632720
in the [test runner execution model][] section.
@@ -667,6 +755,10 @@ test runner functionality:
667755
* `--test-reporter` - Reporting is managed by the parent process
668756
* `--test-reporter-destination` - Output destinations are controlled by the parent
669757
* `--experimental-config-file` - Config file paths are managed by the parent
758+
* `--test-randomize` - Randomization is managed by the parent process and
759+
propagated to child processes
760+
* `--test-random-seed` - Randomization seed is managed by the parent process and
761+
propagated to child processes
670762

671763
All other Node.js options from command line arguments, environment variables,
672764
and configuration files are inherited by the child processes.
@@ -1575,6 +1667,14 @@ changes:
15751667
that specifies the index of the shard to run. This option is _required_.
15761668
* `total` {number} is a positive integer that specifies the total number
15771669
of shards to split the test files to. This option is _required_.
1670+
* `randomize` {boolean} Randomize execution order for test files and queued tests.
1671+
This option is not supported with `watch: true`.
1672+
**Default:** `false`.
1673+
* `randomSeed` {number} Seed used when randomizing execution order. If this
1674+
option is set, runs can replay the same randomized order deterministically,
1675+
and setting this option also enables randomization. The value must be an
1676+
integer between `0` and `4294967295`.
1677+
**Default:** `undefined`.
15781678
* `rerunFailuresFilePath` {string} A file path where the test runner will
15791679
store the state of the tests to allow rerunning only the failed tests on a next run.
15801680
see \[Rerunning failed tests]\[] for more information.

‎doc/node-config-schema.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -524,6 +524,14 @@
524524
"type": "boolean",
525525
"description": "run tests with 'only' option set"
526526
},
527+
"test-random-seed": {
528+
"type": "number",
529+
"description": "seed used to randomize test execution order"
530+
},
531+
"test-randomize": {
532+
"type": "boolean",
533+
"description": "run tests in a random order"
534+
},
527535
"test-reporter": {
528536
"oneOf": [
529537
{
@@ -906,6 +914,14 @@
906914
"type": "boolean",
907915
"description": "run tests with 'only' option set"
908916
},
917+
"test-random-seed": {
918+
"type": "number",
919+
"description": "seed used to randomize test execution order"
920+
},
921+
"test-randomize": {
922+
"type": "boolean",
923+
"description": "run tests in a random order"
924+
},
909925
"test-reporter": {
910926
"oneOf": [
911927
{

‎doc/node.1‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1363,6 +1363,22 @@ tests must satisfy \fBboth\fR requirements in order to be executed.
13631363
Configures the test runner to only execute top level tests that have the \fBonly\fR
13641364
option set. This flag is not necessary when test isolation is disabled.
13651365
.
1366+
.It Fl -test-random-seed
1367+
Set the seed used to randomize test execution order.
1368+
This applies to both test file execution order and queued tests within each file.
1369+
Providing this flag enables randomization implicitly, even without
1370+
\fB--test-randomize\fR.
1371+
The value must be an integer between 0 and 4294967295.
1372+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
1373+
.
1374+
.It Fl -test-randomize
1375+
Randomize test execution order.
1376+
This applies to both test file execution order and queued tests within each file.
1377+
This can help detect tests that rely on shared state or execution order.
1378+
The seed used for randomization is printed in the test summary and can be
1379+
reused with \fB--test-random-seed\fR.
1380+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
1381+
.
13661382
.It Fl -test-reporter
13671383
A test reporter to use when running tests. See the documentation on
13681384
test reporters for more details.
@@ -2040,6 +2056,10 @@ one is included in the list below.
20402056
.It
20412057
\fB--test-reporter-destination\fR
20422058
.It
2059+
\fB--test-randomize\fR
2060+
.It
2061+
\fB--test-random-seed\fR
2062+
.It
20432063
\fB--test-reporter\fR
20442064
.It
20452065
\fB--test-rerun-failures\fR

‎lib/internal/test_runner/runner.js‎

Lines changed: 73 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ const {
5959
validateObject,
6060
validateOneOf,
6161
validateInteger,
62+
validateUint32,
6263
validateString,
6364
validateStringArray,
6465
} = require('internal/validators');
@@ -84,6 +85,7 @@ const {
8485
const { FastBuffer } = require('internal/buffer');
8586

8687
const {
88+
createRandomSeed,
8789
convertStringToRegExp,
8890
countCompletedTest,
8991
kDefaultPattern,
@@ -105,12 +107,14 @@ const kIsolatedProcessName = Symbol('kIsolatedProcessName');
105107
const kFilterArgs = [
106108
'--test',
107109
'--experimental-test-coverage',
110+
'--test-randomize',
108111
'--watch',
109112
'--experimental-default-config-file',
110113
];
111114
const kFilterArgValues = [
112115
'--test-reporter',
113116
'--test-reporter-destination',
117+
'--test-random-seed',
114118
'--experimental-config-file',
115119
];
116120
const kDiagnosticsFilterArgs = ['tests', 'suites', 'pass', 'fail', 'cancelled', 'skipped', 'todo', 'duration_ms'];
@@ -168,6 +172,8 @@ function getRunArgs(path, { forceExit,
168172
argv: suppliedArgs,
169173
execArgv,
170174
rerunFailuresFilePath,
175+
randomize,
176+
randomSeed,
171177
root: { timeout },
172178
cwd }) {
173179
const processNodeOptions = getOptionsAsFlagsFromBinding();
@@ -209,6 +215,12 @@ function getRunArgs(path, { forceExit,
209215
if (rerunFailuresFilePath) {
210216
ArrayPrototypePush(runArgs, `--test-rerun-failures=${rerunFailuresFilePath}`);
211217
}
218+
if (randomize) {
219+
ArrayPrototypePush(runArgs, '--test-randomize');
220+
}
221+
if (randomSeed != null) {
222+
ArrayPrototypePush(runArgs, `--test-random-seed=${randomSeed}`);
223+
}
212224

213225
ArrayPrototypePushApply(runArgs, execArgv);
214226

@@ -646,6 +658,8 @@ function run(options = kEmptyObject) {
646658
lineCoverage = 0,
647659
branchCoverage = 0,
648660
functionCoverage = 0,
661+
randomize: suppliedRandomize,
662+
randomSeed: suppliedRandomSeed,
649663
execArgv = [],
650664
argv = [],
651665
cwd = process.cwd(),
@@ -674,6 +688,56 @@ function run(options = kEmptyObject) {
674688
if (globPatterns != null) {
675689
validateArray(globPatterns, 'options.globPatterns');
676690
}
691+
if (suppliedRandomize != null) {
692+
validateBoolean(suppliedRandomize, 'options.randomize');
693+
}
694+
if (suppliedRandomSeed != null) {
695+
validateUint32(suppliedRandomSeed, 'options.randomSeed');
696+
}
697+
let randomize = suppliedRandomize;
698+
let randomSeed = suppliedRandomSeed;
699+
700+
if (randomSeed != null) {
701+
randomize = true;
702+
}
703+
if (watch) {
704+
if (randomSeed != null) {
705+
throw new ERR_INVALID_ARG_VALUE(
706+
'options.randomSeed',
707+
randomSeed,
708+
'is not supported with watch mode',
709+
);
710+
}
711+
if (randomize) {
712+
throw new ERR_INVALID_ARG_VALUE(
713+
'options.randomize',
714+
randomize,
715+
'is not supported with watch mode',
716+
);
717+
}
718+
}
719+
if (rerunFailuresFilePath) {
720+
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
721+
// TODO(pmarchini): Support rerun-failures with randomization by
722+
// persisting the randomization seed in the rerun state file.
723+
if (randomSeed != null) {
724+
throw new ERR_INVALID_ARG_VALUE(
725+
'options.randomSeed',
726+
randomSeed,
727+
'is not supported with rerun failures mode',
728+
);
729+
}
730+
if (randomize) {
731+
throw new ERR_INVALID_ARG_VALUE(
732+
'options.randomize',
733+
randomize,
734+
'is not supported with rerun failures mode',
735+
);
736+
}
737+
}
738+
if (randomize) {
739+
randomSeed ??= createRandomSeed();
740+
}
677741

678742
validateString(cwd, 'options.cwd');
679743

@@ -683,10 +747,6 @@ function run(options = kEmptyObject) {
683747
);
684748
}
685749

686-
if (rerunFailuresFilePath) {
687-
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
688-
}
689-
690750
if (shard != null) {
691751
validateObject(shard, 'options.shard');
692752
// Avoid re-evaluating the shard object in case it's a getter
@@ -783,11 +843,18 @@ function run(options = kEmptyObject) {
783843
functionCoverage: functionCoverage,
784844
cwd,
785845
globalSetupPath,
846+
randomize,
847+
randomSeed,
786848
};
849+
787850
const root = createTestTree(rootTestOptions, globalOptions);
788851
let testFiles = files ?? createTestFileList(globPatterns, cwd);
789852
const { isTestRunner } = globalOptions;
790853

854+
if (randomize) {
855+
root.diagnostic(`Randomized test order seed: ${randomSeed}`);
856+
}
857+
791858
if (shard) {
792859
testFiles = ArrayPrototypeFilter(testFiles, (_, index) => index % shard.total === shard.index - 1);
793860
}
@@ -833,6 +900,8 @@ function run(options = kEmptyObject) {
833900
rerunFailuresFilePath,
834901
env,
835902
workerIdPool: isolation === 'process' ? workerIdPool : null,
903+
randomize,
904+
randomSeed,
836905
};
837906

838907
if (isolation === 'process') {

0 commit comments

Comments
 (0)