fix(ast): bound extractor traversal by heap, not the interpreter stack

[cdeust]

Problem

Every full-tree AST walker in the extractor layer spent one Python frame per AST level. On a source nested deeper than the interpreter's recursion limit, they raised RecursionError — and nothing catches it:

$ grep -rn "RecursionError\|setrecursionlimit" mcp_server/
(no matches)

ast_parser.py catches ImportError and DownloadError only, so the error propagated out of the indexer and failed the run for the entire repository, not just the offending file.

Reproduced before the fix:

recursionlimit: 1000
nesting=  500 ast_depth= 503 -> OK
nesting= 1000 ast_depth=1003 -> RecursionError: maximum recursion depth exceeded

This is reachable in normal operation. Cortex indexes third-party repositories, and minified or generated sources nest far deeper than hand-written code. It is not reachable from the current first-party corpus: I parsed 400 JS files in cortex-viz and none hit the limit. So this is a latent robustness defect on untrusted input, not an active outage.

Scope

The defect was not one function. A sweep for self-recursive walkers found nine, and seven of them descend the whole tree:

Function	Module	Converted
_walk_type	ast_extractors.py	yes
_walk_for_calls	ast_extractors.py	yes
_walk_java	ast_extractors_jvm.py	yes
_walk_kotlin	ast_extractors_jvm.py	yes
_walk_csharp	ast_extractors_clike.py	yes
_walk_ruby	ast_extractors_scripting.py	yes
_walk_php	ast_extractors_scripting.py	yes
_extract_swift_node	ast_extractors_extra.py	yes
_extract_js_node	ast_extractors.py	no — see below

_extract_js_node recurses only through export_statement, so its depth is bounded by export nesting rather than tree depth. Converting it would be churn without a defect behind it, so it is left alone deliberately rather than by omission.

Fixing only _walk_type would have left Java, Kotlin, C#, Ruby, PHP, Swift and Python call-extraction with the identical crash in the same module family. That is the half-implementation §15 forbids.

Approach

Each walker keeps an explicit list[tuple[Node, str]] stack carrying the enclosing class scope, with descendants pushed reversed so they pop before the remaining siblings. That preserves depth-first pre-order, and with it the order of defs and the insertion order of extract_calls_per_function's dict — which a naive breadth-first rewrite would silently change.

Control flow is preserved exactly, including two subtleties:

• In the JVM/clike/scripting walkers the recursive return sat inside if name:, so an unnamed type node fell through to the method check. The iterative form reproduces that with a continue inside if name:, and says so at the site.
• _extract_swift_node is if/elif/else, so a _SWIFT_KIND_MAP node never reaches the catch-all descent and an unnamed one descends nowhere. Preserved.

Behaviour preservation

Unit tests are weak evidence for an order-preserving rewrite, so the primary evidence is a differential harness that runs the old implementations from HEAD and the new ones side by side over the same corpus and compares output, including dict key order.

ast_extractors (Python + JS):   1344 real files    0 mismatches
  compared: extract_calls_per_function (values AND key order),
            _walk_type over 4 node types

six language walkers:            885 sources       0 mismatches
  java 14 · kotlin 403 · c_sharp 9 · ruby 50 · php 7 · swift 402
  (403 Kotlin and 402 Swift are real files from this machine's corpus)

Files where the old code raised RecursionError were excluded from the comparison rather than counted as mismatches, since there is no old output to compare against.

Tests

tests_py/core/test_ast_extractor_depth.py asserts the depth contract for all six language entry points plus extract_calls_per_function. tests_py/core/test_ast_extractors.py::TestWalkType covers _walk_type directly.

Each depth case asserts two properties, because either alone is insufficient:

1. the fixture's AST is genuinely deeper than sys.getrecursionlimit(), so a future parser change that flattens the tree cannot leave the test green while testing nothing;
2. the nesting sits where the walker actually descends.

Property 2 is not theoretical. My first version of this test was vacuous for 8 of 9 cases: the language walkers return at a method node without entering its body, so nesting inside a method body never reached the recursive path. It was caught only by running the new test against the pre-fix code and seeing just one case fail. Every fixture in the committed version was verified to raise RecursionError against the old implementation before being accepted.

pre-fix : 7 failed
post-fix: 7 passed

Completion Ledger

§	Item	Evidence
A1	Happy paths	pytest tests_py -k "ast or extractor or codebase" → 1157 passed, 5968 deselected
A2	Edge cases	Unnamed type node (falls through, preserved + commented); node with no body; _SWIFT_KIND_MAP node that never reaches the catch-all; empty children; no-match search — TestWalkType::test_returns_empty_when_nothing_matches, ::test_includes_the_start_node_itself
A3	Failure paths	The failure path is the subject: RecursionError is no longer reachable by depth. Asserted by 7 cases that fail on pre-fix code.
A4	Trust boundary	Input is untrusted third-party source. Depth is now bounded by heap, not by the C stack.
A5	Invariants	Stated in each docstring: depth-first pre-order preserved; scope carried per node. Verified differentially over 2229 sources.
A6	Idempotency	Pure functions over a parsed tree; no state.
B1–B3	Concurrency	N/A — no concurrency touched; the walkers are pure and hold no shared state.
C1	Scalability	Unchanged O(n) in node count. Heap allocation replaces stack frames; the stack holds at most the frontier, which is bounded by tree width.
C2	Resource lifecycle	No handles, no allocation outside the local list.
C3	Hot path	Not measured. The traversal is the same node count with the same per-node work; the change is where the frame lives. No regression claimed and none measured.
D1–D3	Security	Removes an uncaught-exception denial path on adversarial input: one deeply nested file previously failed the whole repository's index run. No secrets, no injection surface.
E1	API compatibility	All seven are module-private (_-prefixed) except through unchanged public entry points. No signature changed.
E2	Downstream consumers	extract_calls_per_function, extract_{java,kotlin,csharp,ruby,php,swift}_definitions, and the _walk_type importers in four modules. All verified by the differential above, not by inspection.
E3	Persisted data	None.
E4	Cross-platform	Pure Python; no paths, encoding, or process spawning.
F1	Signals	No new failure mode to signal. The removed one was an uncaught exception.
F2	Degraded modes	None introduced.
G1	Path→test ledger	7 converted functions → 7 depth cases + 4 TestWalkType cases + the differential harness.
G2	Regression test fails pre-fix	Yes, quoted above: 7 failed pre-fix, 7 passed post-fix.
G3	Determinism	Suite re-run after formatting: 1157 passed both times.
G4	Negative assertions	test_returns_empty_when_nothing_matches; the depth guard asserts the fixture is deeper than the limit, failing loudly if it stops being so.
G5	Full gate	ruff check . → All checks passed! · ruff format --check . → 1275 files already formatted · pyright mcp_server/ → 0 errors, 0 warnings, 0 informations
H1	Standards	No layer change; core/ still imports only shared/ + stdlib. Function sizes unchanged in order of magnitude and under the 40-line local cap.
H2	Readability	Each converted walker's docstring states why it is iterative and what ordering guarantee the reversed push buys.
H3	Conventions	Matches the module's existing style; one language per file.
H4	CHANGELOG	See the entry in this PR.
H5	Commit hygiene	Logic and formatting in the same commit is avoided: formatting was applied by ruff format to the touched files only.
H6	CI green	To be confirmed on the pushed tree before merge.
H7	Boy-scout (§14)	Sweep for the defect class run across all five extractor modules, not just the reported one; that is how the other six were found. Mutation survivors: see below.

Mutation (§12)

Scoped run over the five changed files, against HEAD as a paired control (same tool, same source scope):

	mutants	killed	no tests	survived
HEAD	1554	475	851	228
this PR	1486	571	492	423

The rise in survived is reclassification, not regression: the new depth tests execute walker code no test previously reached, so mutants moved from "never run" into "run but not detected". The comparable metric is not-killed = survived + no tests, and it improves in every converted function:

function	HEAD	this PR
_walk_for_calls	40	13
_walk_java	59	37
_walk_kotlin	77	54
_walk_csharp	59	36
_walk_ruby	53	33
_walk_php	61	51
_extract_swift_node	19	17
_walk_type	0	0
total	368	241

241 mutants remain unpinned in these walkers. That is inherited debt — the Java, Kotlin, C#, Ruby, PHP and Swift extractors have almost no behavioural tests — reduced 34% by this PR and not introduced by it, so per §12.5 it is not treated as a blocker here. Filed as #369 with the reproduction command and acceptance criteria, per §14.3; not deferred as prose.

A tooling defect found while doing this, also in #369. scripts/mutation_check.sh printed none — 0 surviving mutants 🎉 on this exact run while its own progress counter ended at 🙁 423, and mutmut results returns a 915-line non-empty list. Its grep-based survivor detection produced a false green, and its cleanup trap deletes mutants/ and .mutmut-cache on exit, so the discrepancy is uninvestigable after the fact. Every mutation number quoted above comes from a direct mutmut run + mutmut results with artifacts retained, not from the script's verdict. A mutation gate that can report a false clean is worse than no gate.