t11s

@transmissions11

cs @mit, prev @sailresearchco @paradigm

2λ.com

Joined September 2020

Replies Reposts Media

t11s
@transmissions11
Sep 14
I feel like it's still underdiscussed that seemingly all of the world's ~severe misalignment incidents as of now occurred in evaluation/RL? This is not what LW/MIRI/etc were expecting iiuc. Why aren't we seeing incidents in deployment? Is eval awareness that strong?
20
t11s
@transmissions11
Sep 9
part 2/2 of my work this summer! TPU v6e has insane FLOPs/$ value, but performance out of the box on SGLang-JAX/vLLM-TPU is often lacking So we wrote our own collective matmul kernels in Pallas, pushing Gemma 4 31B prefill performance up to ~63% MFU(!) sailresearch.com/blog/tpu-v6e-g…
Sail Research
@sailresearchco
Sep 9
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput. v6e has 2.5x less HBM and less than half the memory bandwidth of an H100. @transmissions11 wrote it up here: sailresearch.com/blog/tpu-v6e-g…
7
t11s
@transmissions11
Sep 8
Megakernels are sick, but ifl comparing low batch size performance against vLLM is kind of a cop out inference engines are optimized for high throughput & big batches! very different regime either actually compete at large batch size or just flex absolute numbers like MBU imo
4
t11s
@transmissions11
Sep 7
Increasingly worried that safety/etc researchers inside labs will become far more capable than those outside, just from more token spend enabled by margin-free prices Labs should offer similar deals to folks @ METR/etc if they're serious about preventing concentration of power
Lisan al Gaib
@scaling01
Sep 6
Meanwhile the 90th percentile of OpenAI researchers is spending over $7000 a day on coding agents
9
t11s
@transmissions11
Aug 27
prefill tok/$ ≈ FLOPs/$ decode tok/$ ≈ HBM BW/$ but not always! consider a chip with poor interconnect pipeline parallelism can rescue prefill, but decode needs many seqs resident to fill a deep pipeline, eating KV capacity and shrinking per-chip batch size + throughput!
Neil Movva
@neilmovva
Aug 27
many people are asking: how does Sail pick the best chips? we have a few ways of looking at this, which @transmissions11 formalized into a standard model for inference accelerators - now open source! We loved working with him this summer, he's a star :)
12