no chunking on the read side: half of every long Source is invisible to dense retrieval

[jasonssdev]

openkos has no chunking layer on the read side. One document = one embedding = one retrieval unit = one context block. That single decision produces the retrieval defects filed separately as #882 and #887, and one more that is worse than either: roughly half of every long Source is invisible to dense retrieval.

The asymmetry is stark, because the codebase already solved this on the other side: extraction chunks long sources through _fan_out_windows (src/openkos/extraction/concept.py:2287-2295). openkos chunks when it writes knowledge and does not when it reads it.

Proof: the tail of a long document contributes nothing to its own embedding

src/openkos/state/reindex.py:320-330 composes title + description + tags + full body and sends it in one call:

embed_text = _compose_embed_text(metadata, body)
...
vector = embedder.embed([text])[0]

bge-m3 allows 8192 tokens (/api/show: bert.context_length = 8192). bundle/sources/transcription3.md is 56,037 chars ≈ 15,785 tokens. Embedding the whole document, its first half, and its second half, then comparing cosine similarity:

cos(full, FIRST half)  = 1.0000
cos(full, SECOND half) = 0.6582
cos(FIRST, SECOND)     = 0.6582

1.0000, exactly. The full document's embedding is its first half's embedding. The second half contributes nothing — this is absence, not degradation.

Confirmed one vector per document, not per chunk: SELECT count(*) FROM vector_meta = 32 for 32 embeddable docs.

Blast radius in a 3-source corpus

source	chars	~tokens	invisible to dense retrieval
transcription1.md	55,403	15,606	47%
transcription2.md	10,604	2,987	none
transcription3.md	57,116	16,089	49%

The silent asymmetry between the two retrieval halves

query prints retrieval: 10 FTS + 10 dense → 5 fused. FTS5 has no length limit and indexes the same four fields (fts.py:220-234), so the lexical half sees the whole document while the dense half sees a prefix. Demonstrated with terms that occur only in the invisible tail of transcription3:

sqlite3 .openkos/fts.db "SELECT count(*) FROM docs WHERE docs MATCH 'desprende';"  -> 1
sqlite3 .openkos/fts.db "SELECT count(*) FROM docs WHERE docs MATCH 'máquina';"    -> 1

Findable lexically, unrepresentable semantically. Nothing discloses that the two halves of the fusion have different coverage.

Why the symptoms follow

• query sends unbounded context and files provenance the model never read #882 (unbounded context, false provenance): the retrieval unit is a whole 55 KB document, so the prompt is enormous by construction.
• the attribution probe never measured the context-size regime where the line actually drops #887 (attribution probe never measured the context regime): the wild regime only exists because retrieval units are whole documents.
• This issue: half of every long Source is unreachable by semantic search, silently.

Fixing the chunking layer collapses all three. Fixing #882 alone leaves the embedding blindness untouched.

Proposed fix

1. Chunk on index. Split each document into windows below the embedder's limit, embed each, and store (concept_id, chunk_index, char_span) rows instead of one row per document. The window logic can likely reuse what extraction already does.
2. Retrieve chunks, cite parents. A context block becomes the matching chunk rather than the whole file, and provenance records the parent concept plus the span — which is also the honest fix for query sends unbounded context and files provenance the model never read #882's citation problem, since the span names exactly what the model was shown.
3. Budget the assembled context against the chat window. Still needed, but with chunk-sized units it rarely binds, so query sends unbounded context and files provenance the model never read #882's fix becomes cheap rather than a truncation policy.

Alternative considered

Stop embedding the raw source text into bundle/sources/*.md and keep it only in raw/. That removes the huge documents at the root, but changes the bundle format and loses the property that a Source document is self-contained. Chunking is the less invasive root fix and helps every long document, not only Sources.

Note on the measurement

context_window: 12288 in openkos.yaml is a separate lever and does not touch this: the embedder's 8192 limit is bge-m3's own, and raising the chat window changes nothing about what got embedded. Re-running openkos reindex will not help either — it recomputes the same one-vector-per-document truncated embedding.

Repro

python3 - <<'PY'
# embed a >8192-token document whole, then its two halves, and compare cosines
PY

Any bundle/sources/*.md above ~29,000 chars reproduces cos(full, first_half) == 1.0000.

[jasonssdev]

Coupled defect found while drafting the proposal, verified in source — the migration for this issue will trigger it, but it is reachable today.

reindex renders the tag change at cli/main.py:17063-17065 as:

embedding model changed ({previous_model_tag} -> {cfg.embedding_model})

previous_model_tag = db.read_model_tag() is the stored effective tag, and _effective_model_tag (state/reindex.py:87-93) composes it as f"{model_tag}#{EMBED_COMPOSITION_TAG}". The right-hand side is the bare configured model name. So:

• Today, any workspace that has embedded at least once stores bge-m3#compose-v1, and a genuine model change prints bge-m3#compose-v1 -> gemma2:27b — leaking an internal composition token the user never configured.
• Under this issue's migration, bumping EMBED_COMPOSITION_TAG prints embedding model changed (bge-m3#compose-v1 -> bge-m3): a model change nobody made, with an arrow that reads as a downgrade.

Since this change's rollout depends on that exact tag gate, the disclosure fix belongs here rather than in a separate issue. Note tests/unit/cli/test_purge.py:801 asserts the substring "embedding model changed", so rewording is not free.