Wu Lin

@LinYorker

Postdoctoral fellow at @VectorInst. ML PhD at UBC. Mathematical and computational structures for ML. Geometric and algebraic methods.

Orlando Florida

yorkerlin.github.io

Replies Reposts Media

Pinned
Wu Lin
@LinYorker
Jul 16, 2024
#ICML2024 Can We Remove the Square-Root in Adaptive Methods? arxiv.org/abs/2402.03496 Root-free (RF) methods are better on CNNs and competitive on Transformers compared to root-based methods (AdamW) Removing the root makes matrix methods faster: Root-free Shampoo in BFloat16 /1
4
Wu Lin
@LinYorker
Jun 12
On one hand, it is essential to tune baseline methods well on a model. On the other hand, it may be better to avoid using a model/architecture that has been modified and optimized for a single method for 1.5 years.
Konstantin Mishchenko
@konstmish
Jun 11
I just submitted a PR to modded-nanogpt with better hyperparams. With them, Muon can reach the target loss after 3250 steps instead of 3325. Always tune your baseline well when doing research. Weak baselines can make any idea look promising
1
Wu Lin
@LinYorker
May 3
We will make Shampoo/SOAP, including KL-Shampoo/KL-SOAP, faster. Our goal is to match Muon's runtime while maintaining Shampoo/SOAP's strong per-step performance. Stay tuned for new updates.
Lucas Nestler
@Clashluke
May 2
KL Shampoo and KL SOAP outperform their non-KL counterparts by learning the preconditioners compositionally, so that each stage corrects what remains after the last. Available in HeavyBall 3.1.1, with major PSGD stability backports.
1
Wu Lin
@LinYorker
Apr 30
@_arohan_ Muon
1
Wu Lin
@LinYorker
Apr 19
@weijie444 Looks like a KFAC-based method with modern clipping? G(ZZ^T)^{-1} is known as the FOOF update arxiv.org/abs/2201.12250 while msgn() can be interpreted as "generalized (preconditioned) gradient norm clipping" arxiv.org/abs/2506.01913 .
Weijie Su
@weijie444
Apr 16
We released "The Newton--Muon Optimizer" . We show that Muon is secretly an implicit Newton method, and use this insight to build a better one. 1/n Paper: arxiv.org/abs/2604.01472
2