$47.9 million mistake not naming the company 🫶 Heart Hands
- Qwen 27B dense at 105 tok/s output on an m5 max is pretty bonkers. Breaking down the memory wall one brick at a time.We’re releasing speculative decoding in our inference engine Uzu, starting with Qwen3.6-27B. On Apple M5 Max with 128 GB of unified memory, our Mirai-M stack reaches 105 output tokens/sec entirely on-device - 2.9× faster than the fastest MLX speculative-decoding implementation
- Interesting that Apple pays Google ~billion per year for Gemini when they can get a better model for free (GLM 5.2 and soon Kimi K3).
- The video from @angeloskath on local agentic AI with MLX is excellent. I also hear it's one of the most viewed videos in WWDC history 👏 Goes through the basics of agentic AI and how to set it all up to run locally in a very approachable and simple way. The demos are excellent
- It's very cool that Apple shipped a 20B parameter on-device. You can't put 20B parameters in RAM at any reasonable precision. To make it work they are using pretty exotic architecture by today's standards. A small model predicts from the query (or prompt) which experts to load
