Skip to content

test(ast): pin the remaining extractor surface — 639 unpinned → 51 classified equivalents, 0 uncovered (#369) - #376

Merged
cdeust merged 1 commit into
mainfrom
test/369-pin-remaining-extractors
Aug 7, 2026
Merged

test(ast): pin the remaining extractor surface — 639 unpinned → 51 classified equivalents, 0 uncovered (#369)#376
cdeust merged 1 commit into
mainfrom
test/369-pin-remaining-extractors

Conversation

@cdeust

@cdeust cdeust commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Finishes #369. #373 closed the seven scope-tracking walkers and named the rest as a tail; a named tail is still an unfinished PR.

Result

at filing after #373 now
unpinned mutants, five modules 639 594 51
mutants no test executes 462 462 0
killed 847 892 1435 of 1486

Zero uncovered is the number that matters most. Every mutant in these modules is now executed by a test; what remains is 51 that run and are not detected, and each of those has an argument rather than a shrug.

What is covered

file scope
test_ast_extractor_imports.py all eleven import extractors
test_ast_extractor_definitions.py C, C++, Go, Rust, Python, JS/TS definitions + callee basename
test_ast_extractor_edges.py fields, constants at their boundary, character sets

The third file exists because shape assertions structurally cannot see three classes of drift, which only mutation exposed:

  • ImportInfo.names was asserted nowhere. The resolver binds a reference to a definition through those names, so the whole name-collection path was free to change silently.
  • Constants at their boundary — the [:120] signature cut and len(base) < 100. No ordinary fixture separates 120 from 121, or < from <=.
  • Character setsrstrip(";"), strip('"<>'), replace("using", "", 1). An ordinary module path cannot tell the correct set from a wider one.

Hence the deliberately awkward fixtures: paths ending in X, a callee named maXker, a name of exactly 100 characters, a[b[c]](), (a or b)(), and bytes that are not valid UTF-8. Each makes one specific drift observable and nothing else does.

Two are worth reading as behaviour rather than bookkeeping. The Ruby scanner test pins that a puts between two requires is skipped rather than ending the scan — turning that skip into a break would silently drop every import after the first unrelated call, which in Ruby is most of them. And _text decoding invalid bytes matters because Cortex indexes whatever is on disk; a broken error handler raises LookupError out of the indexer on the first non-UTF-8 file, and valid input never consults it.

Seven rough edges, now pinned as current behaviour

Each with its cause stated, so changing one takes a failing test. Each is a plausible resolution defect. None is fixed here — that is a separate decision per item, and I would rather you pick than have me quietly change extraction semantics inside a test PR.

behaviour cause
C emits a struct twice when a typedef names it two _walk_type passes, no dedup
C++ drops struct S {} while C keeps it C++ looks for class_specifier only
Rust drops functions inside mod m {} no descent into module blocks
Rust keeps use a::{B, C} and use x as y verbatim no group expansion, no alias strip; neither string resolves
Python drops import os.path as p the alias wraps the dotted_name, so it is not a direct child
JS ignores require() and new Klass() ES imports only; new_expression is not a call node
PHP group use App\{A, B} loses its prefix A and B are emitted bare and are unresolvable

The 51 survivors, classified

Two categories, and the distinction matters for how long each claim lasts.

Provably equivalent by construction — no fixture can ever distinguish them:

  • rsplit(sep, n)[-1] is the text after the last separator for every n; split(sep, n)[0] the text before the first. Only direction matters, and direction is pinned by a[b[c]]().
  • None vs "" as an initial scope, at seven call sites. Both falsy; every use is a truthiness test.
  • _declarator_name's two if branches have identical bodies, so which tuple a node type sits in is unobservable.

Unreachable today — verified by reading the callers, not by failing to find a fixture. These can become false if a grammar or a caller changes, which is why they are labelled differently:

  • _extract_js_node's method_definition arm is only reached from _extract_js_class (line 227), which always supplies a class name, so its else cannot run. "function" is a legacy node type; the grammar now emits function_expression.
  • Go: function_declaration always has parameters, method_declaration always has a receiver.
  • Python: function_definition always has a name and parameters; class_definition always has a name. The superclasses fallback is not dead, which is why class C: pass is a test.

Kept with a filed follow-up: the decorated_definition arm, #372.

Source changes are documentation only

Proved rather than asserted — the parsed AST of each changed file, docstrings stripped, is identical to HEAD:

mcp_server/core/ast_extractors.py         AST identical ignoring docstrings: True
mcp_server/core/ast_extractors_clike.py   AST identical ignoring docstrings: True
mcp_server/core/ast_extractors_extra.py   AST identical ignoring docstrings: True

Completion Ledger

§ Item Evidence
A1 Happy paths pytest -k "ast or extractor or codebase"1265 passed
A2 Edge cases Empty source for all eleven import extractors; unnamed constructs; nested types; boundary-length names; non-UTF-8 bytes; fragments with no enclosing type
A3 Failure paths No new failure path; the decode-handler path is now exercised for the first time
A4 Trust boundary Fixtures include invalid UTF-8 and syntactically partial sources, which is what Cortex indexes
A5 Invariants Ordering asserted, not just membership — several extractors emit in pass order, not source order, and consumers read it positionally
A6 Idempotency Pure functions over a parsed tree
B1–B3 Concurrency N/A
C1–C3 Resources / perf N/A — tests and comments only
D1–D3 Security N/A
E1 API compatibility No behaviour change; AST-identical proof above
E2 Downstream consumers Unchanged by construction
E3–E4 Persisted data, cross-platform N/A
F1–F2 Observability N/A
G1 Path→test ledger Every public entry point in the five modules now has an exact-output test; enumerated by grep -nE '^def [a-z]' per module and checked off
G2 Regression tests N/A as a bugfix; each round's fixtures were verified to move the mutation count (639 → 99 → 58 → 52 → 51) rather than assumed to
G3 Determinism Suite run five times across the change; 1265 each time
G4 Negative assertions Aliased plain import dropped; require() ignored; new ignored; Rust mod contents absent; Go type alias absent; C++ struct absent; over-cap callee dropped
G5 Full gate ruff check . clean · ruff format --check . 1279 files formatted · pyright mcp_server/ 0 errors, 0 warnings, 0 informations
H1–H3 Standards, readability, conventions Documentation-only source diff; test style matches the existing extractor tests
H4 CHANGELOG N/A — tests and comments, no consumer-observable change
H5 Commit hygiene One commit; formatting applied by ruff format to touched files only
H6 CI green Required before merge
H7 Boy-scout (§14) Seven rough edges surfaced and documented rather than silently frozen; the gate's own limitation raised below

One thing this does not fix

scripts/mutation_check.sh still exits 1, and will keep doing so. It has no notion of an accepted equivalent, so a module whose every survivor carries a written rationale is indistinguishable to it from one nobody has looked at. That makes the blocking per-commit gate unusable on these five files for any future one-line change.

I have not invented a suppression mechanism here, because a badly designed one is worse than none — a wildcard "ignore these" list would re-create the false-green this same script was just fixed for. It needs a typed, registered, fail-closed exemption keyed to a specific mutant with a required rationale. Worth its own issue and its own decision; say the word and I will file it with a proposed design.

Finishes what #373 left. Unpinned mutants across the five extractor modules:
639 -> 51, and mutants no test executes at all: 462 -> 0. Killed 1435 of 1486.

#373 closed the seven scope-tracking walkers and named the rest as a tail. A
named tail is still an unfinished PR, so this covers everything else in those
modules: all eleven import extractors, C, C++, Go, Rust, Python and JS/TS
definitions, and callee-basename resolution.

Three new test modules, all asserting exact ordered output and all measured
against the implementation before being asserted rather than predicted:

  test_ast_extractor_imports.py      the eleven import extractors
  test_ast_extractor_definitions.py  C/C++/Go/Rust/Python/JS + callee basename
  test_ast_extractor_edges.py        fields, constants and character sets

The third exists because shape assertions structurally cannot see three
classes of drift, which mutation made visible:
  - ImportInfo.names was asserted nowhere, so the whole name-collection path
    was free to change silently; the resolver binds references through it
  - constants at their boundary: the [:120] signature cut and the
    len(base) < 100 callee cap. No ordinary fixture separates 120 from 121
  - character sets in string cleanup: rstrip(";"), strip('"<>'),
    replace("using", "", 1). An ordinary module path cannot tell the real set
    from a wider one

Hence deliberately awkward fixtures: paths ending in X, a callee named
maXker, a name of exactly 100 characters, a[b[c]](), (a or b)(), and bytes
that are not valid UTF-8 (Cortex indexes whatever is on disk, and a broken
decode handler would raise LookupError out of the indexer on the first such
file — valid input never consults it).

Seven rough edges are now pinned as CURRENT behaviour with the cause named,
so changing one takes a failing test: C emits a struct twice when a typedef
names it; C++ drops `struct` while C keeps it; Rust drops functions inside
`mod {}` and keeps `use a::{B,C}` / `as` clauses verbatim; Python drops
`import os.path as p`; JS ignores require() and new; PHP group-use loses its
prefix. Each is a plausible resolution defect, none is fixed here.

The 51 survivors are classified, not tolerated, with the argument at the use
site. Provably equivalent: rsplit(sep,n)[-1] and split(sep,n)[0] do not
depend on n; None vs "" as an initial scope (both falsy, every use is a
truthiness test); _declarator_name's two branches have identical bodies.
Unreachable, verified from the callers: _extract_js_node's method arm (only
reached from _extract_js_class, which always supplies a name), Go's params
and receiver fallbacks, Python's name/params fallbacks. Kept with a filed
follow-up: the decorated_definition arm (#372).

Source changes are documentation only — proved by comparing the parsed AST of
each file against HEAD with docstrings stripped: identical for all three.

Verified:
  pytest -k "ast or extractor or codebase"  -> 1265 passed
  ruff check / format --check               -> clean
  pyright mcp_server/                       -> 0 errors, 0 warnings
  scoped mutation                           -> 1435 killed, 0 uncovered, 51 equivalent

Refs #369, #372

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cdeust
cdeust force-pushed the test/369-pin-remaining-extractors branch from e8ebd6a to 2a6a5cc Compare August 7, 2026 07:59
@cdeust
cdeust merged commit cdac77b into main Aug 7, 2026
21 checks passed
@cdeust
cdeust deleted the test/369-pin-remaining-extractors branch August 7, 2026 08:20
cdeust added a commit that referenced this pull request Aug 7, 2026
§12.4 is "0 surviving NON-EQUIVALENT mutants, or each survivor documented as
equivalent". scripts/mutation_check.sh could express the first half and not
the second: any survivor failed, so a module where every survivor carries a
written rationale looked identical to one nobody had read. After #376 the five
extractor modules sit at 50 justified survivors, which made the blocking
per-commit tier permanently red there — and a gate that is permanently red for
a good reason is a gate people learn to bypass.

Adds memory/mutation-equivalents.json plus scripts/mutation_equivalents.py.
Three properties, in priority order:

  Fail-closed on drift. mutmut numbers mutants positionally, so
  x__walk_type__mutmut_16 names a DIFFERENT mutation once the function moves.
  A name-keyed exemption would silently absolve whatever later occupies the
  slot. Every entry pins the exact removed/added pair and applies only while
  the diff still matches; a changed mutation is unregistered and fails.

  Fail-closed on absence. Only named mutants are exempt. No wildcards, no
  per-file suppression, no ignore-module — those are how a suppression
  mechanism becomes the next false green, which this same script produced
  yesterday (it printed "0 surviving mutants" on a 423-survivor run).

  No silent rot. An entry matching no produced mutant is an error: either a
  test now kills it (dead weight) or the code moved (rationale unverified).
  Staleness is scoped to the files a run actually mutated, so a narrow run
  does not flag entries it never exercised.

Ordering in the shell is load-bearing: the registry filters first, then the
#269 recheck re-verifies only what is left. Reversed, every registered
equivalent would burn a full test selection and be reported as genuine.

kind is a closed enum of three because the claims age differently:
equivalent-by-construction is about semantics and cannot become false;
unreachable-branch is about today's callers or grammar and can. rationale has
a minimum length because "equivalent" as a bare word is the thing this exists
to make unwriteable.

The registry carries 50 entries — 27 equivalent-by-construction, 21
unreachable-branch, 2 seam-pending-implementation (#372) — not the 51
survivors #376 left. While writing rationales I could not justify the Ruby
strip("()") mutant, so I tested instead of asserting: require(Xfoo) yields
Xfoo and the widened set yields foo. It was killable, and got a test rather
than an exemption.

Verified:
  gate on the five extractor modules -> EXIT 0
    1436 killed, 0 uncovered, 50 registered equivalents accepted
  negative control, one entry removed -> EXIT 1
    49 accepted, 1 unregistered, recheck confirms 1 GENUINE survivor
  pytest tests_py/scripts/test_mutation_equivalents.py -> 23 passed
  ruff check / format --check -> clean; bash -n -> ok

Refs #369, #372

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant