We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo
TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!
Zachary Charles
@MatharyCharles
distributed machine learning @ google | sometimes mathematician
