Skip to content

mutmut false-survives _make_extractor/build_extra_extractors — eager module-level dispatch table defeats coverage-based test selection #269

Description

@cdeust

What

scripts/mutation_check.sh (mutmut 3.x) reports 50 "survived" mutants for
mcp_server/core/ast_extractor_registry.py's _make_extractor (8 variants)
and build_extra_extractors (42 variants) even when the correct, full test
selection (tests_py/core/test_ast_parser.py +
tests_py/core/test_ast_extractors_multilang.py +
tests_py/core/test_ast_extractors.py) genuinely kills every one of them.
This is a false-survivor report from mutmut's own coverage-based
test-selection optimizer, not a real test gap — verified directly below.

Root cause

mcp_server/core/ast_parser.py builds its language-dispatch table once,
eagerly, at import time:

_EXTRACTORS = {
    "python": _extract_python,
    ...
    **build_extra_extractors(),
}

build_extra_extractors() calls _make_extractor(...) once per language
(java/kotlin/c/cpp/csharp/ruby/php) to build the closures stored in this
dict. mutmut's coverage-based test selection (mutants/mutmut-stats.json,
tests_by_mangled_function_name) records, per mutant, which test(s)
called the mutated function during a baseline coverage pass — but
because _EXTRACTORS is built exactly once, the very first test in
collection order to import mcp_server.core.ast_parser is the only test
mutmut ever sees calling _make_extractor/build_extra_extractors
directly. Every other test that exercises the SAME dict entries later
(java/kotlin/c/cpp/csharp/ruby/php parsing) does so through the
already-built, cached dict — never re-invoking the trampoline-wrapped
functions — so mutmut's stats never attribute those tests as "covering"
the mutant, and its scoped re-run for that mutant selects only the one
(irrelevant, Python-language) test that happened to trigger the eager
import.

Reproduction (2026-07-29, macOS, mutmut 3.x, worktree building on #249)

$ python3 -m mutmut run   # scoped to ast_parser.py + ast_extractor_registry.py
...  🙁 50 survived (all in _make_extractor / build_extra_extractors)

$ python3 -c "import json; d=json.load(open('mutants/mutmut-stats.json')); \
    print(d['tests_by_mangled_function_name']['mcp_server.core.ast_extractor_registry.x__make_extractor'])"
['tests_py/core/test_ast_parser.py::TestParseFilePython::test_returns_file_analysis']

# Direct verification the mutant IS actually caught by the real, full suite:
$ MUTANT_UNDER_TEST="mcp_server.core.ast_extractor_registry.x__make_extractor__mutmut_1" \
    python3 -m pytest tests_py/core/test_ast_parser.py \
    tests_py/core/test_ast_extractors_multilang.py \
    tests_py/core/test_ast_extractors.py -q
7 failed, 50 passed   # test_java, test_kotlin, test_c, test_cpp, test_csharp, test_ruby, test_php

All 8 _make_extractor mutants and all 42 build_extra_extractors mutants
were verified this way (looping MUTANT_UNDER_TEST over every mutant name
against the full three-file test selection): every one fails or errors.
Zero are real survivors.

Why it matters

scripts/mutation_check.sh / tools/mutation_check.sh back the
per-commit blocking mutation gate (coding-standards.md §12.1/§12.5). A
false "survived" report on genuinely-covered code either (a) blocks a
correct commit until someone manually re-derives that the survivor is
fake (as this issue required), or worse (b) trains reviewers to wave off
survived-mutant reports as "probably a tooling artifact" without
verifying — reintroducing exactly the silent-failure risk §12 exists to
close. Any module that builds a dispatch table eagerly at import time
(a common, otherwise-good pattern) is structurally invisible to mutmut's
per-mutant test-selection optimizer.

Acceptance criteria

  1. scripts/mutation_check.sh (or its mutmut config) produces zero false
    survivors for mcp_server/core/ast_extractor_registry.py's
    _make_extractor/build_extra_extractors — either by forcing a full
    (non-stats-restricted) test run for mutants in modules with eager
    module-level dispatch-table construction, or by another mechanism that
    closes the gap described above.
  2. The fix (or a documented, cited limitation) is verified against this
    issue's exact reproduction: re-running mutmut afterward shows 0
    survivors for these two functions, OR the tool's own output makes the
    false-survivor cause visible to the person reading the report (so it
    is never silently trusted as a real gap).
  3. No regression to scripts/mutation_check.sh's existing scoped-run
    behavior for modules that do NOT build eager module-level dispatch
    tables (the committed json_native.py scope stays green).

Related

#262 (mutmut cannot attribute mutants for scripts/ modules — a
different root cause: synthetic import module names vs path-derived
ones). This issue's root cause is eager import-time memoization of a
dispatch table, not module-naming.

Found during #249 (typecheck resolver-dependency fix), while running
scripts/mutation_check.sh-equivalent scoped mutation testing on
mcp_server/core/ast_parser.py and mcp_server/core/ast_extractor_registry.py
(dead-code removal + Swift/Rust/Go test-coverage backfill in that PR). The
50 survivors are NOT deferred untreated: this issue exists because I
already did the verification work above and confirmed they are false
positives, not because the verification is left undone.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions