Evan Miller

@EvMill

Statistically inclined software developer, occasional blogger about math + stats stuff. Working @AnthropicAI

NYC

evanmiller.org

Replies Reposts Media

Pinned
Evan Miller
@EvMill
Jul 24, 2023
I hit a bug in the Attention formula that’s been overlooked for 8+ years. All Transformer models (GPT, LLaMA, etc) are affected. Researchers isolated the bug last month – but they missed a simple solution… Why LLM designers should stop using Softmax 👇
evanmiller.org
Attention Is Off By One
Let’s fix these pesky Transformer outliers using Softmax One and QuietAttention.
74
Evan Miller
@EvMill
Aug 26, 2024
New sequential A/B test from @Zalando based on the Lévy inequality – check it out!
arxiv.org
YEAST: Yet Another Sequential Test
Large-scale randomised experiments have become a standard tool for developing products and improving user experience. To reduce losses from shipping harmful changes experimental results are, in...
Evan Miller
@EvMill
Aug 5, 2024
I think I've finally cracked quantiles… A/B testing medians, instead of means, usually requires an expensive bootstrap. But we can use a likelihood-ratio test (Wilks' theorem) instead. This reduces the quantile problem to a few simple formulas. Read on!
arxiv.org
Likelihood-ratio inference on differences in quantiles
Quantiles can represent key operational and business metrics, but the computational challenges associated with inference has hampered their adoption in online experimentation. One-sample...
Evan Miller
@EvMill
Sep 25, 2023
👀
Markus Nagel
@mnagel87
Sep 25, 2023
Replying to @Tracing47202686 @yell1337 and @TiRune
Unlike with clipped softmax, to achieve an exact zero in the output using softmax1 for a (partial) no-update, the input requires to be -infinity. However, after @EvMill blog post we experimented with softmax1 and found it in practice competitive with our proposed approaches.
1
Evan Miller
@EvMill
Aug 29, 2023
Softmax1 update… We now have support for ⚡️Flash Attention ⚡️ This lets us test much larger models than before! To get the code, just pip install flash-attention-softmax-n Or clone / star the GitHub repo here: github.com/softmax1/Flash… All credit / kudos to Chris Murphy.
GitHub - softmax1/Flash-Attention-Softmax-N: CUDA and Triton implementations of Flash Attention...
From github.com
1