jevbetter

Train a small model that chooses among a changing list of text options, done better.

jevbetter is a one-pass option scorer (type: text in, one probability per option out, single forward pass, no token-by-token decoding), built from scratch with a stronger encoder, a rival-aware scoring head, and a sharper training loop. Same JSONL data format as the open-source jevlike scorer, so any jevlike dataset trains here unchanged, and the benchmark below compares the two head-to-head on identical data.

Independent project. Not affiliated with TypeSafe or the Jev model.

A Jev-like model takes a piece of text and a list of N text options and returns one probability per option, in one pass instead of writing an answer word by word. That makes it a natural fit for routing, ranking, classification with open label sets, and game controllers.

jevbetter-benchmark trains both models on the identical hard synthetic dataset with matched budgets and prints the comparison:

$ jevbetter-benchmark --reference /path/to/jevlike --epochs 8

800 held-out hard menus (2–8 options, near-miss negatives, distractor sentences), CPU, same 8 epochs / batch size / seed. jevbetter wins on accuracy (+4.3pp top-1) and calibration (2× lower ECE); the shuffled-context control scores 0.335 top-1, confirming the model genuinely reads the context. The transformer context encoder costs throughput: 40 menus/sec is still orders of magnitude faster than a decoder writing hundreds of tokens, and --context-features tunes the tradeoff.

python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'

jevbetter-data --output data/synthetic          # hard synthetic menus
jevbetter-train data/synthetic/train.jsonl \
  --validation data/synthetic/validation.jsonl \
  --output runs/model.pt
jevbetter-eval runs/model.pt data/synthetic/test.jsonl
jevbetter-predict runs/model.pt \
  --context "Choose the exact badge amber badger. Badge: amber badger." \
  --option "azure crane" \
  --option "amber badger" \
  --option "gold heron"

The evaluation prints top-1/3/5 accuracy, MRR, calibration error, accuracy by menu size, throughput, and a shuffled-context control (each menu paired with the wrong context, which a useful model beats comfortably).

JSONL, one object per line (drop-in compatible with jevlike):

{"context": "The customer needs a refund.", "options": ["refund", "sales", "technical support"], "label": 0}

Or CSV with context,options,label columns (options separated by |). label is the zero-based index of the correct option; minimum two options per row, no duplicates.

Swap the n-gram encoder for a frozen pretrained transformer under the same rival-aware scoring head:

pip install -e '.[transformers]'
jevbetter-train data/synthetic/train.jsonl \
  --validation data/synthetic/validation.jsonl \
  --output runs/qwen-head.pt \
  --encoder hf \
  --hf-model Qwen/Qwen2.5-0.5B

The checkpoint stores the trained head and the encoder name, not the frozen weights, so loading needs access to the same Hugging Face model.

Each option's text becomes a mean-pooled n-gram vector. Options first attend to each other (rival-aware), then each queries the transformer-encoded context with multi-head attention. The option↔context interaction passes through a sigmoid gate into a two-layer MLP that emits one score; a temperature-scaled softmax over options gives the probabilities.

• This is a research starter, not a copy of any commercial model. We make no claim about matching TypeSafe's Jev or its private training method.
• One-pass scoring needs the complete option list before prediction.
• The n-gram encoder is stronger than byte embeddings but still shallow next to a pretrained transformer: use --encoder hf when meaning matters most.
• Accuracy depends on data quality and split quality, as always.

Code is MIT. Pretrained models you download keep their own terms.