Pinned
I found out the other day that any compression tool can be contorted to do language modeling. Turns out gzip can generate text that somewhat *resembles* Shakespeare. Short write up linked below
Nathan Barry
@nathanrs
Descending the gradient
- LLMs can be near-perfect lossless compressors via arithmetic coding, yet nobody uses them for compression. Found out why: ran an 0.8B model on 1MB of text. It compressed 2x better than gzip, but ran ~70,000x slower (1 hour vs 0.05s)
- New post! A drawback of diffusion LLMs is that bidirectional attention causes keys and values to drift across steps, breaking KV caching. But generation quality is robust to slight KV drift, and much work has gone into maximizing stale KV reuse without causing quality drops (1/3)
- I noticed that Apple Notes has a similar UI as the AI chat apps, so I turned it into a Claude/ChatGPT frontend. Use any LLM API to interact or chat with in Apple Notes
- Diffusion LLMs are becoming very competitive architectures. But recently, there's also been a lot of progress in flow-based LLMs, which are conceptually similar. Both learn to transport samples from a noise distribution to a data distribution. Image generation used to beMercury 2 is live 🚀🚀 The world’s first reasoning diffusion LLM, delivering 5x faster performance than leading speed-optimized LLMs. Watching the team turn years of research into a real product never gets old, and I’m incredibly proud of what we’ve built. We’re just getting
