Log inSign up
user avatar
Jonathan Hayase
@JonathanHayase
Final year ML PhD student at UW CSE
jon.jon.ke
Joined June 2020
160
Following
249
Followers
RepliesMedia
  • user avatar
    Jonathan Hayase
    @JonathanHayase
    Jul 15, 2025
    Do you ever wish all LLMs used the same tokenizer?🧑‍🤝‍🧑 We present an *efficient, lossless* method to convert any LM into a byte-level model at inference time. This fixes weird tokenization artifacts at the prompt boundary and enables ensembles of LMs with mismatched tokenizers! 🧵
    Image
  • user avatar
    Jonathan Hayase
    @JonathanHayase
    Mar 21, 2025
    Tokenizers govern the allocation of computation. It's a waste to spend a whole token of compute predicting the "way" in "By the way". SuperBPE redirects that compute to predict more difficult tokens, leading to wins on downstream tasks!
    user avatar
    Alisa Liu
    @alisawuffles
    Mar 21, 2025
    We created SuperBPE🚀, a *superword* tokenizer that includes tokens spanning multiple words. When pretraining at 8B scale, SuperBPE models consistently outperform the BPE baseline on 30 downstream tasks (+8% MMLU), while also being 27% more efficient at inference time.🧵
    Segmentation of the sentence "By the way, I am a fan of the Milky Way" under BPE and SuperBPE.
  • user avatar
    Jonathan Hayase
    @JonathanHayase
    Jul 29, 2024
    Tokenizers and autogregressive LMs are both trained to compress text, but tokenizer training is deterministic and we know exactly how it works! This makes inverse problems wrt the data much easier. There's a wealth of info lurking in public tokenizers waiting to be extracted!