SOAP on @kellerjordan0's nanogpt benchmark. SOAP seems to take ~> 10% less steps as compared to OrthogonalNesterov optimizer but also has more overhead.
github.com/nikhilvyas/mod…
I will be at ICML from Monday onwards presenting icml.cc/virtual/2024/p…, icml.cc/virtual/2024/p… and icml.cc/virtual/2024/3…. I have recently been thinking about optimization (linking some recent works below) but have pretty broad interests and would love to chat with people.
1/n A technical thread on our results in arxiv.org/pdf/2406.17748 on connecting the Shampoo optimizer and Optimal Kronecker product approximation of the the Adagrad (or Hessian) preconditioner.
Why does Shampoo work well? Our new work sheds light on this, highlighting a wide misconception about the optimizer. We show the *square* of Shampoo's preconditioner is provably near to the optimal Kronecker approximation of the (Adagrad) Hessian. See:
arxiv.org/pdf/2406.17748
