test(ast): pin the remaining extractor surface — 639 unpinned → 51 classified equivalents, 0 uncovered (#369) - #376
Merged
Conversation
Finishes what #373 left. Unpinned mutants across the five extractor modules: 639 -> 51, and mutants no test executes at all: 462 -> 0. Killed 1435 of 1486. #373 closed the seven scope-tracking walkers and named the rest as a tail. A named tail is still an unfinished PR, so this covers everything else in those modules: all eleven import extractors, C, C++, Go, Rust, Python and JS/TS definitions, and callee-basename resolution. Three new test modules, all asserting exact ordered output and all measured against the implementation before being asserted rather than predicted: test_ast_extractor_imports.py the eleven import extractors test_ast_extractor_definitions.py C/C++/Go/Rust/Python/JS + callee basename test_ast_extractor_edges.py fields, constants and character sets The third exists because shape assertions structurally cannot see three classes of drift, which mutation made visible: - ImportInfo.names was asserted nowhere, so the whole name-collection path was free to change silently; the resolver binds references through it - constants at their boundary: the [:120] signature cut and the len(base) < 100 callee cap. No ordinary fixture separates 120 from 121 - character sets in string cleanup: rstrip(";"), strip('"<>'), replace("using", "", 1). An ordinary module path cannot tell the real set from a wider one Hence deliberately awkward fixtures: paths ending in X, a callee named maXker, a name of exactly 100 characters, a[b[c]](), (a or b)(), and bytes that are not valid UTF-8 (Cortex indexes whatever is on disk, and a broken decode handler would raise LookupError out of the indexer on the first such file — valid input never consults it). Seven rough edges are now pinned as CURRENT behaviour with the cause named, so changing one takes a failing test: C emits a struct twice when a typedef names it; C++ drops `struct` while C keeps it; Rust drops functions inside `mod {}` and keeps `use a::{B,C}` / `as` clauses verbatim; Python drops `import os.path as p`; JS ignores require() and new; PHP group-use loses its prefix. Each is a plausible resolution defect, none is fixed here. The 51 survivors are classified, not tolerated, with the argument at the use site. Provably equivalent: rsplit(sep,n)[-1] and split(sep,n)[0] do not depend on n; None vs "" as an initial scope (both falsy, every use is a truthiness test); _declarator_name's two branches have identical bodies. Unreachable, verified from the callers: _extract_js_node's method arm (only reached from _extract_js_class, which always supplies a name), Go's params and receiver fallbacks, Python's name/params fallbacks. Kept with a filed follow-up: the decorated_definition arm (#372). Source changes are documentation only — proved by comparing the parsed AST of each file against HEAD with docstrings stripped: identical for all three. Verified: pytest -k "ast or extractor or codebase" -> 1265 passed ruff check / format --check -> clean pyright mcp_server/ -> 0 errors, 0 warnings scoped mutation -> 1435 killed, 0 uncovered, 51 equivalent Refs #369, #372 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cdeust
force-pushed
the
test/369-pin-remaining-extractors
branch
from
August 7, 2026 07:59
e8ebd6a to
2a6a5cc
Compare
cdeust
added a commit
that referenced
this pull request
Aug 7, 2026
§12.4 is "0 surviving NON-EQUIVALENT mutants, or each survivor documented as equivalent". scripts/mutation_check.sh could express the first half and not the second: any survivor failed, so a module where every survivor carries a written rationale looked identical to one nobody had read. After #376 the five extractor modules sit at 50 justified survivors, which made the blocking per-commit tier permanently red there — and a gate that is permanently red for a good reason is a gate people learn to bypass. Adds memory/mutation-equivalents.json plus scripts/mutation_equivalents.py. Three properties, in priority order: Fail-closed on drift. mutmut numbers mutants positionally, so x__walk_type__mutmut_16 names a DIFFERENT mutation once the function moves. A name-keyed exemption would silently absolve whatever later occupies the slot. Every entry pins the exact removed/added pair and applies only while the diff still matches; a changed mutation is unregistered and fails. Fail-closed on absence. Only named mutants are exempt. No wildcards, no per-file suppression, no ignore-module — those are how a suppression mechanism becomes the next false green, which this same script produced yesterday (it printed "0 surviving mutants" on a 423-survivor run). No silent rot. An entry matching no produced mutant is an error: either a test now kills it (dead weight) or the code moved (rationale unverified). Staleness is scoped to the files a run actually mutated, so a narrow run does not flag entries it never exercised. Ordering in the shell is load-bearing: the registry filters first, then the #269 recheck re-verifies only what is left. Reversed, every registered equivalent would burn a full test selection and be reported as genuine. kind is a closed enum of three because the claims age differently: equivalent-by-construction is about semantics and cannot become false; unreachable-branch is about today's callers or grammar and can. rationale has a minimum length because "equivalent" as a bare word is the thing this exists to make unwriteable. The registry carries 50 entries — 27 equivalent-by-construction, 21 unreachable-branch, 2 seam-pending-implementation (#372) — not the 51 survivors #376 left. While writing rationales I could not justify the Ruby strip("()") mutant, so I tested instead of asserting: require(Xfoo) yields Xfoo and the widened set yields foo. It was killable, and got a test rather than an exemption. Verified: gate on the five extractor modules -> EXIT 0 1436 killed, 0 uncovered, 50 registered equivalents accepted negative control, one entry removed -> EXIT 1 49 accepted, 1 unregistered, recheck confirms 1 GENUINE survivor pytest tests_py/scripts/test_mutation_equivalents.py -> 23 passed ruff check / format --check -> clean; bash -n -> ok Refs #369, #372 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finishes #369. #373 closed the seven scope-tracking walkers and named the rest as a tail; a named tail is still an unfinished PR.
Result
Zero uncovered is the number that matters most. Every mutant in these modules is now executed by a test; what remains is 51 that run and are not detected, and each of those has an argument rather than a shrug.
What is covered
test_ast_extractor_imports.pytest_ast_extractor_definitions.pytest_ast_extractor_edges.pyThe third file exists because shape assertions structurally cannot see three classes of drift, which only mutation exposed:
ImportInfo.nameswas asserted nowhere. The resolver binds a reference to a definition through those names, so the whole name-collection path was free to change silently.[:120]signature cut andlen(base) < 100. No ordinary fixture separates 120 from 121, or<from<=.rstrip(";"),strip('"<>'),replace("using", "", 1). An ordinary module path cannot tell the correct set from a wider one.Hence the deliberately awkward fixtures: paths ending in
X, a callee namedmaXker, a name of exactly 100 characters,a[b[c]](),(a or b)(), and bytes that are not valid UTF-8. Each makes one specific drift observable and nothing else does.Two are worth reading as behaviour rather than bookkeeping. The Ruby scanner test pins that a
putsbetween tworequires is skipped rather than ending the scan — turning that skip into a break would silently drop every import after the first unrelated call, which in Ruby is most of them. And_textdecoding invalid bytes matters because Cortex indexes whatever is on disk; a broken error handler raisesLookupErrorout of the indexer on the first non-UTF-8 file, and valid input never consults it.Seven rough edges, now pinned as current behaviour
Each with its cause stated, so changing one takes a failing test. Each is a plausible resolution defect. None is fixed here — that is a separate decision per item, and I would rather you pick than have me quietly change extraction semantics inside a test PR.
_walk_typepasses, no dedupstruct S {}while C keeps itclass_specifieronlymod m {}use a::{B, C}anduse x as yverbatimimport os.path as pdotted_name, so it is not a direct childrequire()andnew Klass()new_expressionis not a call nodeuse App\{A, B}loses its prefixAandBare emitted bare and are unresolvableThe 51 survivors, classified
Two categories, and the distinction matters for how long each claim lasts.
Provably equivalent by construction — no fixture can ever distinguish them:
rsplit(sep, n)[-1]is the text after the last separator for everyn;split(sep, n)[0]the text before the first. Only direction matters, and direction is pinned bya[b[c]]().Nonevs""as an initial scope, at seven call sites. Both falsy; every use is a truthiness test._declarator_name's twoifbranches have identical bodies, so which tuple a node type sits in is unobservable.Unreachable today — verified by reading the callers, not by failing to find a fixture. These can become false if a grammar or a caller changes, which is why they are labelled differently:
_extract_js_node'smethod_definitionarm is only reached from_extract_js_class(line 227), which always supplies a class name, so itselsecannot run."function"is a legacy node type; the grammar now emitsfunction_expression.function_declarationalways hasparameters,method_declarationalways has a receiver.function_definitionalways has a name and parameters;class_definitionalways has a name. Thesuperclassesfallback is not dead, which is whyclass C: passis a test.Kept with a filed follow-up: the
decorated_definitionarm, #372.Source changes are documentation only
Proved rather than asserted — the parsed AST of each changed file, docstrings stripped, is identical to
HEAD:Completion Ledger
pytest -k "ast or extractor or codebase"→ 1265 passedgrep -nE '^def [a-z]'per module and checked offrequire()ignored;newignored; Rustmodcontents absent; Go type alias absent; C++ struct absent; over-cap callee droppedruff check .clean ·ruff format --check .1279 files formatted ·pyright mcp_server/0 errors, 0 warnings, 0 informationsruff formatto touched files onlyOne thing this does not fix
scripts/mutation_check.shstill exits 1, and will keep doing so. It has no notion of an accepted equivalent, so a module whose every survivor carries a written rationale is indistinguishable to it from one nobody has looked at. That makes the blocking per-commit gate unusable on these five files for any future one-line change.I have not invented a suppression mechanism here, because a badly designed one is worse than none — a wildcard "ignore these" list would re-create the false-green this same script was just fixed for. It needs a typed, registered, fail-closed exemption keyed to a specific mutant with a required rationale. Worth its own issue and its own decision; say the word and I will file it with a proposed design.