Skip to content

Test runner's --test-rerun-failures could swallow actual failures on retries #63424

Description

@atlowChemi

Version

26.1.0

Platform

Darwin ARM64

Subsystem

test_runner

What steps will reproduce the bug?

The following example test file will reproduce the bug:

// Demonstrates that `node --test --test-rerun-failures` can mark a failing test as passing on retry without executing its body.
// Root cause: the runtime disambiguator at lib/internal/test_runner/test.js:793-801 keys tests purely by `file:line:column`.
// A `t.test()` registered inside a factory function gets the same source location regardless of which parent invoked the factory.
// The disambiguator counter is assigned in execution order across the whole run, so on retry - when the set of running tests changes,
// the surviving failing test inherits a counter slot that, in attempt 0, belonged to a different (passing) sibling.
// Node matches by that slot, replaces `this.fn` with a synthetic noop replay, and reports the failure as a pass.

import { describe, it } from 'node:test';
import { strict as assert } from 'node:assert';

function makeSuite(shouldPass, label) {
  return async (t) => {
    await t.test('inner', async () => {
      if (!shouldPass) assert.fail(`${label} should fail`);
    });
  };
}

describe('parents', { concurrency: false }, () => {
  it('A passes', makeSuite(true,  'A'));
  it('B fails',  makeSuite(false, 'B'));   // the only real failure
  it('C passes', makeSuite(true,  'C'));
});

Executed with:

node --test --test-rerun-failures=rerun.json repro.mjs   # attempt 0
# B fails as expected; rerun.json now contains A and C plus their
# inner subtests keyed as "repro.mjs:13:13" and "repro.mjs:13:13:(1)"

node --test --test-rerun-failures=rerun.json repro.mjs   # retry
# all 6 tests reported as passing, exit code 0
# B's assert.fail was never executed

How often does it reproduce? Is there a required condition?

Happens consistently

What is the expected behavior? Why is that the expected behavior?

The test inner should not be replaced with a noop for B fails, allowing it to actually execute

What do you see instead?

The inner test of B fails is skipped completely, due to a erroneous disambiguation

Additional information

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmed-bugIssues and PRs for confirmed bugs.test_runnerIssues and PRs related to the test runner subsystem.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions