Overview
Kimetsu's house rule is that every claim ships with a measurement.
Kimetsu's house rule is that every claim ships with a measurement. This section documents how we measure the brain and what the numbers are, so you can check them rather than take our word for it.
The headline numbers
The v2.8.0 structured-answerability study reports a separate paired experiment: unwanted injections 15/18 → 3/18, with 24/27 positive hits retained and exact evidence metadata matching 36/45 cases per repeat. P95 changed from 376.6 to 386.6 ms; mean response size from 613.5 to 651.2 bytes. The linked study records build provenance and release status. It does not replace the historical published scores below or remeasure them on v2.8.0.
| benchmark | result |
|---|---|
| LoCoMo | 89.4% (1,540 questions, LLM-judged accuracy) |
| BEAM 100K | 73.3%, matching the prior public state of the art, model-free |
| BEAM 1M | 66.0% (300 probes, 15 of 35 conversations) |
LongMemEval (_s) | 83.0% (200-question stratified slice) |
| BrainBench quality index | 80.0% (142 scenarios, reader-free) |
| retrieval quality | recall@4 0.949, MRR 0.914 at ~138 ms |
| stale-hit rate | 0.091 (was 0.500 on flat retrieval) |
| cost per solved task | ~13x cheaper than a no-brain baseline |
The historical results in the table use jina-v2-base-code + the ms-marco-tinybert-l-2-v2
cross-encoder reranker unless noted, and the memory pipeline makes zero LLM
calls: the reader in the public benchmarks answers questions, it never stores
or retrieves.
How this section is organized
We measure on three layers, one page each plus the comparison:
- Retrieval and correctness: the in-repo
bench that gates every release. Recall, MRR, latency, stale-hit rate, and
contradiction resolution, runnable with
kimetsu brain bench. - BrainBench: our own reader-free capability benchmark. It drives the real binary across difficulty tiers and scores dedup, forgetting, importance, and calibration, the write-path behaviour a reader-driven test can't see.
- Public benchmarks, directly comparable to other memory systems: LongMemEval (chat-domain, per-question-type) and BEAM (ten memory abilities over long multi-session chats).
- How Kimetsu compares: the honest side-by-side against mem0, Cognee, Zep, and Letta, including where they lead.
Kimetsu Remote
Run the brain on a server and connect over HTTP MCP, so a team (or you across machines) shares one brain per repository, with no local checkout.
Retrieval & correctness
Kimetsu's retrieval and correctness numbers: recall, MRR, latency, stale-hit suppression, and contradiction resolution, all reproducible from the shipped CLI.
