Jonathan Hayase

@JonathanHayase

Final year ML PhD student at UW CSE

jon.jon.ke

Joined June 2020

Replies Reposts Media

Jonathan Hayase
@JonathanHayase
Aug 16
Motif 3 is using a SuperBPE tokenizer! Very happy to see that superwords are boosting encoding efficiency (=> inference speed) in Korean as well as important domains like math and code!
Motif Technologies
@motif_tech
Aug 11
Motif 3 is officially here. Today, we’re releasing Motif 3 Base and Motif 3. Motif 3 Base is our pretrained foundation model, while Motif 3 is built on top of it with post-training using NVIDIA NeMo-RL. Both models were developed with the support of Korea's Ministry of Science
Jonathan Hayase
@JonathanHayase
Jul 15, 2025
Do you ever wish all LLMs used the same tokenizer?🧑‍🤝‍🧑 We present an *efficient, lossless* method to convert any LM into a byte-level model at inference time. This fixes weird tokenization artifacts at the prompt boundary and enables ensembles of LMs with mismatched tokenizers! 🧵
3
Jonathan Hayase
@JonathanHayase
Mar 21, 2025
Tokenizers govern the allocation of computation. It's a waste to spend a whole token of compute predicting the "way" in "By the way". SuperBPE redirects that compute to predict more difficult tokens, leading to wins on downstream tasks!
Alisa Liu
@alisawuffles
Mar 21, 2025
We created SuperBPE🚀, a *superword* tokenizer that includes tokens spanning multiple words. When pretraining at 8B scale, SuperBPE models consistently outperform the BPE baseline on 30 downstream tasks (+8% MMLU), while also being 27% more efficient at inference time.🧵
1
Jonathan Hayase
@JonathanHayase
Jul 29, 2024
Tokenizers and autogregressive LMs are both trained to compress text, but tokenizer training is deterministic and we know exactly how it works! This makes inverse problems wrt the data much easier. There's a wealth of info lurking in public tokenizers waiting to be extracted!
Alisa Liu
@alisawuffles
Jul 26, 2024
What do BPE tokenizers reveal about their training data?🧐 We develop an attack🗡️ that uncovers the training data mixtures📊 of commercial LLM tokenizers (incl. GPT-4o), using their ordered merge lists! Co-1⃣st @JonathanHayase arxiv.org/abs/2407.16607 🧵⬇️