Whitespace-tokenized retrieval pipelines commonly lose recall on languages that do not use spaces between words. Research across 8 academic papers (MMTEB, XRAG, MIT 2025) describes a broader 5-layer semantic loss cascade:

babel-memory is the fix for layers 2-4. Same simple API. Zero required dependencies. Install only the language packs you need.

BEFORE babel-memory:
  Store: "机器学习在自然语言处理中的应用"
  BM25 search("机器学习") → [] (zero results)
  KG extract → English prompt struggles with Chinese entities
  
AFTER babel-memory:
  Store: "机器学习在自然语言处理中的应用"
         → fts_text: "机器 学习 机器学习 自然 语言 处理 自然语言 应用"
  BM25 search("机器学习") → [match found!]
  KG extract → Chinese prompt with CJK few-shot examples

BEFORE babel-memory (European):
  Store: "Maschinelles Lernen verbessert die Verarbeitung"
  BM25 search("Verarbeitung") → [match]
  BM25 search("verarbeitungen") → [] (different form, zero results)

AFTER babel-memory + snowball-stemmers:
  Store: "Maschinelles Lernen verbessert die Verarbeitung"
         → fts_text: "maschinell lern verbess verarbeit"
  BM25 search("verarbeitungen") → stem("verarbeitungen") = "verarbeit" → [match found!]

# Core only (zero dependencies — pure TypeScript)
npm install babel-memory

# Add language packs as needed:
npm install jieba-wasm          # Chinese (highest quality)
npm install @sglkc/kuromoji     # Japanese (highest quality)
npm install wordcut             # Thai
npm install snowball-stemmers   # 16 mapped algorithms currently usable by this library
npm install tinyld              # auto-detect Latin-script languages (de/fr/es/...)

You only pay for what you use. The core package has zero dependencies — language packs are loaded lazily at runtime. And thanks to the built-in Intl.Segmenter tier (ICU dictionaries shipped with Node 16+/Bun/Deno/browsers), a zero-dependency install already gets word-level segmentation for Chinese, Japanese and Thai — installing the packs upgrades quality further. It never crashes.

import { detectLanguage, initTokenizer, tokenizeForFts, getKgPrompt } from "babel-memory";

// 1. Initialize once at startup (loads whichever packages are installed)
await initTokenizer();

// 2. Detect language (zero dependencies, pure Unicode analysis)
detectLanguage("这个项目的架构设计非常优秀");  // "zh"
detectLanguage("東京タワーはとても高いです");    // "ja" (not "zh" — hiragana detected first)
detectLanguage("이 프로젝트는 매우 훌륭합니다"); // "ko"
detectLanguage("สวัสดีครับ");                    // "th"
detectLanguage("مرحبا بالعالم");                 // "ar"
detectLanguage("Машинное обучение");              // "ru"

// 3. Pre-tokenize for BM25 (the core fix)
tokenizeForFts("机器学习很有趣", "zh");
// → "机器 学习 很 有趣"  (jieba word segmentation)

tokenizeForFts("東京タワー", "ja");
// → "東京 タワー"  (kuromoji word segmentation)

tokenizeForFts("Maschinelles Lernen", "de");
// → "maschinell lern"  (Snowball stemming)

// 4. ⚠️ CRITICAL: apply the SAME tokenization to your queries.
// Tokenization only matches when index side and query side agree:
const ftsQuery = tokenizeForFts(userQuery, detectLanguage(userQuery));
// store side:  "机器学习很有趣" → "机器 学习 很 有趣"  (indexed)
// query side:  "机器学习"       → "机器 学习"          (matches!)
// forgetting this = querying one giant token "机器学习" against
// segmented text = zero results. This is THE classic FTS pre-tokenization trap.

// 5. Get language-matched prompts for LLM calls
const { system, userTemplate } = getKgPrompt("zh"); // → Chinese prompt
getKgPrompt("ja"); // → native Japanese prompt
getKgPrompt("ko"); // → English prompt (instructions in a third language hurt quality)
// Predicates stay English (normalized keys) in every variant

// 6. Optional power tools
import { detectLanguageDetailed, detectLanguageExtended, getLoadedTokenizers } from "babel-memory";

detectLanguageDetailed("我在用 TypeScript 给 RecallNest 写 tokenizer hook");
// → { language: "en", scripts: { cjk: 0.13, latin: 0.83, ... }, isMixed: true }
// AI conversation logs mix CN/EN constantly — isMixed tells you when.
// Segmentation-needing scripts (CJK/Thai) flag mixed from a ~2% ratio:
// even a tiny Chinese island is unsearchable if left unsegmented.
// (Embedded CJK runs are segmented automatically either way; see below.)

detectLanguageExtended("Das ist ein guter Tag");
// → "de" when tinyld is installed; "en" otherwise (graceful degrade)

tokenizeForFts("机器学习的应用", "zh", { removeStopwords: true });
// → "机器 学习 应用"  (的 removed; opt-in, default off)

getLoadedTokenizers(); // → ["jieba", "kuromoji", "tinyld"] — verify deployment

// Selective init when you don't need everything (kuromoji alone = ~1-2s + ~17MB):
await initTokenizer({ languages: ["zh", "de"] });

The key insight: pre-tokenize non-whitespace-delimited text before FTS indexing, and stem inflected languages.

Standard FTS pipeline (broken for Chinese):
  "知识图谱提取" → whitespace split → ["知识图谱提取"] → 1 giant token → no matches

babel-memory pipeline (fixed):
  "知识图谱提取" → jieba segmentation → "知识 图谱 知识图谱 提取" → whitespace split → 4 tokens → matches!

This works with any whitespace-based FTS engine: Tantivy (LanceDB), SQLite FTS5, Elasticsearch, Meilisearch. No engine modifications needed.

bun bench/recall-benchmark.ts — SQLite FTS5, 20 Chinese documents, 12 queries:

On this fixed 20-document / 12-query Chinese fixture, raw text returns no results while both preprocessing tiers return all expected documents. This is a regression smoke test, not a general retrieval-quality benchmark.

Real agent conversations constantly mix languages: "I fixed 机器学习模型 using TensorFlow". Ratio-based detection lands on zh or en depending on the balance — and either route now handles the mix correctly: the zh path's tokenizers keep Latin tokens intact, and the en path detects embedded CJK/Hangul/Thai runs and routes each run to its proper tokenizer. Before v2.1, a Chinese island inside en-classified text passed through unsegmented and unsearchable. Use detectLanguageDetailed() if you want the mixing signal explicitly.

Japanese uses kanji (CJK characters). Naive CJK detection would misclassify Japanese as Chinese. babel-memory checks language-unique scripts first:

1. Hiragana/Katakana present? → Japanese (unique to Japanese)
2. Hangul present? → Korean (unique to Korean)
3. Thai script? → Thai
4. Arabic script? → Arabic
5. Devanagari? → Hindi
6. Cyrillic? → Russian
7. CJK Ideographs without Japanese/Korean markers? → Chinese
8. Default → English

babel-memory never crashes due to a missing optional package. Each language has a three-tier fallback chain — and since v2.1 the middle tier (ICU via Intl.Segmenter, built into Node 16+/Bun/Deno) gives zero-dependency installs word-level quality:

Why bigrams? ICU emits whole compounds ("東京タワー" as one token), which breaks partial-match queries like "タワー". CJK tokens of length ≥ 3 get bigram expansion on both index and query side — the same approach lunr-languages uses.

Why is Korean syllable-level? Korean is agglutinative: "프로젝트는" = "프로젝트" + topic particle. Word-level tokens make the query "프로젝트" miss entirely; syllable split keeps BM25 partial matching stable.

A warning is logged once per missing package so you know what to install for better quality. Your application keeps working regardless.

Type: Language = "zh" | "ja" | "ko" | "th" | "ar" | "hi" | "ru" | "en"

tokenizeForFts also accepts any Snowball language code (e.g., "de", "fr", "es") as a string.

Note on script-based detection limits: detectLanguage cannot distinguish languages that share the Latin alphabet — German, French and Spanish all return "en". That's why detectLanguageExtended + the optional tinyld pack exist: install it and Latin-script languages resolve automatically, completing the auto-detect → Snowball stemming chain. Traditional Chinese note: ICU and jieba both handle zh-Hant text, but jieba's dictionary is Simplified-centric; for mixed Simplified/Traditional corpora consider normalizing externally (OpenCC) before indexing.

Total: 22 distinct supported language codes: 8 auto-detected codes plus 14 additional explicit Snowball codes. Arabic and Russian are already included in the auto-detected eight.

With the optional tinyld pack installed, detectLanguageExtended() can route the supported Latin-script languages into their mapped Snowball algorithms without the caller supplying a code.

• AI memory system builders — if you're building on LanceDB, ChromaDB, or any vector+BM25 hybrid store
• RAG pipeline developers — if your users speak non-English languages and BM25 returns empty
• MCP server authors — if your memory tools need multilingual support
• Anyone who's noticed their AI agent "forgets" non-English conversations

babel-memory is not a memory system — it's a preprocessing layer that makes any memory system work properly across languages.

• RecallNest — MCP-native shared memory for Claude Code, Codex, and Gemini CLI

This library is informed by findings from:

• MMTEB: Massive Multilingual Text Embedding Benchmark (arXiv 2502.13595)
• XRAG: Cross-lingual Retrieval-Augmented Generation (arXiv 2505.10089)
• MIT: Tokenization Changes Meaning in LLMs (Computational Linguistics, 2025)

MIT