Observed (A/B bench rev.2, B-Cortex probe session, 2026-08-07)
In a probe session answering documentation questions about the Cortex repo itself, the document store served verbatim-real but superseded doc snapshots:
- README content matching revision
f80bf539 (2026-06-11): "26 mechanisms · 49 tools" — live HEAD says "36 · 52".
- CLAUDE.md content matching v2.6.0 (2026-03-29): layer counts 11/108/21/33/4 — measured live: 26/230/98/138/0;
server/ is now empty (viz stack moved to cortex-viz).
- The "108 modules" figure was explicitly retracted by
docs/module-inventory.md 9 days before the probe — the store still served it.
- Storage default answered as PostgreSQL; live default is SQLite (PRIVACY.md).
Zero fabrication — every claim traced to a real historical revision. The failure mode is a time-lagged mirror presented as current.
Evidence: harness-comparison bench, results/COMPARISON-rev2.md (B-Cortex verdict detail + error-pattern section) and results/probes/B-Cortex.json. The code-graph outage in that session (†) is unrelated: the doc store was reachable and answered.
Root cause (policy gap, not a bug in any one function)
Doc snapshots captured at ingest time (ingest_codebase / seeding) are stored as memories with no source-revision stamp and no invalidation or re-verification on subsequent ingests of the same repo. Recall then serves them indistinguishably from current facts.
Expected — any of these closes the gap, in order of strength
- Re-verify/re-ingest on
ingest_codebase: when re-ingesting a repo, diff stored doc-derived memories against the live files they came from; supersede (version-chain) the stale ones.
- Provenance stamping at serve time: memories derived from repo files carry
{source_path, revision/mtime, captured_at} and recall surfaces it, so a consumer can see "captured 2026-06-11" next to a 2026-08 question.
- Staleness scoring:
core/staleness.py (file-reference staleness) already exists — wire it into the recall path for doc-derived memories.
Related: #359 (wiki rot — same disease, wiki surface), #345 (write-time provenance hints), #368 (no provenance dimension in WRRF ranking). This issue is the doc-snapshot/memory instance with hard bench evidence.
Why it matters
This was the single dominant error pattern of the ai-architect.tools harness across the whole bench (stale counts on 3 of 4 repos, stale line numbers, one retracted figure re-served). Fixing the freshness policy converts most of the harness's "partial" verdicts to "correct" — it is the highest-leverage retrieval-quality fix identified by the comparison.
Observed (A/B bench rev.2, B-Cortex probe session, 2026-08-07)
In a probe session answering documentation questions about the Cortex repo itself, the document store served verbatim-real but superseded doc snapshots:
f80bf539(2026-06-11): "26 mechanisms · 49 tools" — live HEAD says "36 · 52".server/is now empty (viz stack moved to cortex-viz).docs/module-inventory.md9 days before the probe — the store still served it.Zero fabrication — every claim traced to a real historical revision. The failure mode is a time-lagged mirror presented as current.
Evidence: harness-comparison bench,
results/COMPARISON-rev2.md(B-Cortex verdict detail + error-pattern section) andresults/probes/B-Cortex.json. The code-graph outage in that session (†) is unrelated: the doc store was reachable and answered.Root cause (policy gap, not a bug in any one function)
Doc snapshots captured at ingest time (
ingest_codebase/ seeding) are stored as memories with no source-revision stamp and no invalidation or re-verification on subsequent ingests of the same repo. Recall then serves them indistinguishably from current facts.Expected — any of these closes the gap, in order of strength
ingest_codebase: when re-ingesting a repo, diff stored doc-derived memories against the live files they came from; supersede (version-chain) the stale ones.{source_path, revision/mtime, captured_at}and recall surfaces it, so a consumer can see "captured 2026-06-11" next to a 2026-08 question.core/staleness.py(file-reference staleness) already exists — wire it into the recall path for doc-derived memories.Related: #359 (wiki rot — same disease, wiki surface), #345 (write-time provenance hints), #368 (no provenance dimension in WRRF ranking). This issue is the doc-snapshot/memory instance with hard bench evidence.
Why it matters
This was the single dominant error pattern of the ai-architect.tools harness across the whole bench (stale counts on 3 of 4 repos, stale line numbers, one retracted figure re-served). Fixing the freshness policy converts most of the harness's "partial" verdicts to "correct" — it is the highest-leverage retrieval-quality fix identified by the comparison.