Tri Dao

@tri_dao

Asst. Prof @PrincetonCS, Chief Scientist @togethercompute. Machine learning & systems.

Stanford, CA

tridao.me

Replies Reposts Media

Pinned
Tri Dao
@tri_dao
Jul 11, 2024
FlashAttention is widely used to accelerate Transformers, already making attention 4-8x faster, but has yet to take advantage of modern GPUs. We’re releasing FlashAttention-3: 1.5-2x faster on FP16, up to 740 TFLOPS on H100 (75% util), and FP8 gets close to 1.2 PFLOPS! 1/
35
Tri Dao
@tri_dao
Sep 22
Mayank put in a crazy amount of work to get pretraining to work on 3 gens of Nvidia GPUs and 2 gens of TPUs! Very good model for such a small size
Mayank Mishra
@MayankMish98
Sep 22
We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵
5
Tri Dao
@tri_dao
Sep 1
A lot more gpus coming for open models
Together AI
@togethercompute
Aug 31
We just signed one of the largest AI infrastructure deals for open source, period. 250MW data center. $5B+ in annualized revenue. Built with @HUMAIN in Saudi Arabia. tinyurl.com/4nzcjcu7
12
Tri Dao
@tri_dao
Aug 18
These guys move fast, 1st rack already ships. More inference compute is always welcome
Etched
@Etched
Aug 18
We've raised $700M at a $21B valuation from Jane Street, Kleiner Perkins, Sequoia, A16Z, Peter Thiel, BCV, and Blackstone. We're also excited to share that we've shipped our first rack to Jane Street.
00:00
9
Tri Dao
@tri_dao
Aug 12
I really like this new benchmark. Has the flavor of ARC-AGI3 but it’s pure text so you don’t have to worry about the vision capability
James Whittington
@jcrwhittington
Aug 12
We’re excited to announce DiG-bench, a new benchmark for discovery! Over the last few weeks we’ve been testing frontier AI models on our novel discovery games and seeing how they score. Each game is a text-based environment, so they probe discovery capabilities in the natural
1