Pinned
Cool! We derived an AdamW batch-size scaling rule in Lemma 3.13 of arxiv.org/abs/2411.15958
B → B × k
𝗟𝗥 → 𝗟𝗥 × sqrt(𝗸)
𝗪𝗗 → 𝗪𝗗 × sqrt(𝗸)
βᵢ → 1 − sqrt(k) × (1 − βᵢ)
This was subsequently validated empirically in an ICLR 2026 paper:
openreview.net/pdf?id=elB9k4n…
I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition.
First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓
