fix(wiki): strip quoted/duplicated-label frontmatter scalar values - #104
Merged
Conversation
wiki.pages.title was corrupted for 62 rows (verified via `title LIKE 'title:%' OR title LIKE '"%"'` against the production Cortex DB): core.wiki_pages.parse_page's scalar branch never stripped surrounding YAML quote characters (36 rows, e.g. `title: "Public API surface: automatised-pipeline"` stored WITH the literal quotes) and never guarded against a duplicated `<key>: ` label an LLM-authored page's own frontmatter echoed into its value (26 rows, e.g. on-disk `title: title: "Public API surface: automatised-pipeline"`, confirmed against the real file for id 913). Both patterns flow into wiki.pages via wiki_migrate.page_row_from_md, the same path wiki_write's synchronous sync and the batch wiki_migrate sweep both use. Reproduced both corruption signatures as failing tests against the pre-fix parser, then fixed parse_page's scalar branch via a new _clean_scalar_value helper (dup-label strip, then quote strip) mirroring the quote-stripping the block-list and inline-list branches already do. Included a one-shot backfill script for the 61 rows that predate this fix (title-only fix; upsert_page's UPDATE is body_hash-gated so a plain wiki_migrate re-run silently skips derivation-only corrections) — CSV backup taken before any writes, migration is idempotent, verified 0 rows remain matching the corruption signature post-migration. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Jul 14, 2026
Closed
cdeust
added a commit
that referenced
this pull request
Jul 14, 2026
…_governed_page (#109) write_governed_page persisted caller-supplied markdown verbatim without re-deriving or validating frontmatter -- how the `title: title: "..."` corruption (repaired reactively at read-time by commit 53712df/PR #104) got onto disk in the first place. Adds a write-time gate (core/wiki_frontmatter_validation.py) that round-trips every full-page write through parse_page + render_page so persisted bytes are always canonical, closing the corruption class instead of known signatures. Only the one shape parse_page cannot repair -- an unclosed frontmatter fence, which would silently swallow the body into misparsed keys -- is rejected (raises, becomes a ToolError via safe_handler, the repo's standing idiom). append mode's content is a fragment, not a full page, and is never normalized. Headless-authoring call sites in consolidation/page_io.py catch the new exception explicitly so a malformed page can't crash the batch drain cycle. Co-authored-by: Claude <noreply@anthropic.com>
cdeust
added a commit
that referenced
this pull request
Jul 14, 2026
…dening Cut immediately after v4.14.0 to carry a fix that landed on main just after that tag: - fix(mcp): safe_handler raises ToolError from handler failure instead of returning a schema-violating error dict, so a failing MCP tool call surfaces its real diagnostic instead of the generic "Output validation error" every client previously saw regardless of cause (49f29e9, c7dfc24). - fix(wiki): strip quoted/duplicated-label frontmatter scalar values that broke FS↔PG sync (#104, 53712df). - fix(wiki): wiki_migrate --dry-run made transactionally neutral (#108, b60d268). - feat(wiki): write_governed_page validates and normalizes frontmatter at write time instead of only at read time (#109, 3eb43f2). - refactor(wiki): wiki_pages.py extracted into cohesive collaborators, no behaviour change (#111, 28145f0). Marketplace: cortex 4.14.0 -> 4.14.1 (metadata + pin + version_note). Pre-tag guard (exact tree): LongMemEval MRR 0.9167/R@10 0.982 (matches v4.14.0 0.9166/0.982); LoCoMo 3-run mean MRR 0.7984/R@10 0.9142 (v4.14.0 was 0.8005/0.9131 — delta MRR -0.0021 is smaller than v4.14.0's own documented +-0.0025 single-run variance, not a regression; R@10 improved); BEAM 0.5406 (3 runs, spread 0.0001) vs v4.14.0's 0.5471. BEAM deviation investigated: same-day A/B + bisect + control rerun (9 runs total) show an intra-day production-DB variance band of 0.5391-0.5445 spanned by BOTH trees — a same-day v4.14.0 control run measured 0.5445, then on rerun ~2.5h later 0.5391 (SAME code, SAME tree), and a bisect at 53712df/49f29e98 landed inside that band with no monotonic step at any commit boundary. Code effect excluded on two independent grounds: (1) the bench import closure (BenchmarkDB -> mcp_server.core.memory_ingest + mcp_server.core.pg_recall) is untouched by any 4.14.1 commit — the full mcp_server diff between v4.14.0 and this tag is confined to wiki_*, handlers/consolidation/ page_io.py, handlers/wiki_migrate.py, handlers/wiki_write.py, and tool_error_handler.py, none of which is in that import graph; (2) the SAME v4.14.0 tree reproduces both the high and low ends of the band on rerun. BEAM is not gated by reproduce.sh's floor check (proxy metric, within-system comparison only) so this does not block the tag. Floor-gate result (reproduce.sh::check_floors logic, applied directly to the JSONs since no runner prints it itself; tolerance 0.005): LongMemEval R@10 0.9820 vs floor 0.9820 PASS; LongMemEval MRR 0.9167 vs floor 0.9140 PASS; LoCoMo R@10 0.9142 vs floor 0.9150 PASS; LoCoMo MRR 0.7984 vs floor 0.8050 FAIL (-0.0066, exceeds the 0.005 tolerance) — flagged, not hidden: fails the codebase's hardcoded absolute floor while passing the release's actual stopping rule (no regression beyond noise vs the v4.14.0 baseline). The absolute floor has not been re-baselined since the 4.11-4.13 measurement era and LoCoMo MRR has trended down release-over-release with no corresponding retrieval-code change; recommend recalibrating the floor in a future session against an isolated benchmark DB rather than the live growing production store used here. Reranker active. Results committed under benchmarks/results/repro/20260714-v4.14.1-pretag/ (11 files: 5 primary runs, beam-100K-rep2/rep3, and 4 investigation runs from the bisect: beam-control-v4140, beam-bisect-53712df8, beam-bisect-49f29e98, beam-bisect-da3f2104-rep2). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (proven on disk)
parse_page's scalar branch (mcp_server/core/wiki_pages.py:115-118) assignedraw_strippedverbatim, so:title: "Public API surface: automatised-pipeline") kept its literal quote characters — the list branches already stripped quotes, the scalar branch never did.write_governed_pagecontain a duplicated label on disk (title: title: "..."— confirmed on ids 913 and 2916);line.partition(":")only strips the firsttitle:, so the second rode intowiki.pages.title.Fix
New
_clean_scalar_value(key, raw_stripped)helper: strips one duplicated<key>:label (exact token match, case-insensitive —titleist ...is untouched) then one matching pair of surrounding quotes, mirroring the existing list-branch behavior.Tests
3 new tests: 2 reproductions anchored to the real on-disk corruption (quoted scalar, duplicated label quoted + unquoted variants) and 1 negative (no over-stripping of a value that merely starts with the key name).
tests_py/core/test_wiki_pages.py: 12/12 green; full suite 5102/5102 green; ruff check + format clean.scripts/backfill_wiki_titles_2026_07_14.pyhas already been run against prod (2026-07-14): 62 corrupted rows matched the signature (title LIKE 'title:%'ORtitle LIKE '"%"'), 61 repaired + 1 already fixed incidentally; post-check via psql = 0 remaining. CSV backup of (id, rel_path, title) was taken beforehand. The script is committed for provenance/reproducibility — do not re-run.Rationale for a dedicated script:
upsert_page's UPDATE is gated onbody_hash <> EXCLUDED.body_hash, which correctly skips unchanged files — but the corruption was in the derivation, not the file, so re-runningwiki_migratesilently skips exactly the affected rows.Known follow-ups (not in this PR)
wiki_migrate --dry-runactually commits its passes 1-3 despite the name (footgun, separate fix)wiki_pages.pyat 452 lines exceeds the 500-target trajectory (pre-existing debt → refactorer)write_governed_page🤖 Generated with Claude Code
https://claude.ai/code/session_01HM6rjMN4oHHmL6qPwv6ZPN