Adam depends on the gradient distribution during training, which, as far as I know, we don't understand well?
Here, adapted from the Adam paper, v_t is the var estimate, G_t is the gradient r.v. and X_t is an error r.v. for distribution shift.
Jason Rader
@packquickly
- Screw it, here's a new JAX implementation in Optimsitix: github.com/packquickly/sc…Schedule-Free paper is up! arxiv.org/abs/2405.15682 Joint work with collaborators @alicey_ang @HarshMeh1a @konstmish @akhaledv2 @AshokCutkosky We have some strong small-scale experiments on Transformers, comparing to chinchilla-style cosine 10x reduction schedules.
- ⭐ Lineax is now on arXiv! ⭐ If you’re doing linear solves or linear least-squares in JAX, give it a shot today! Lineax is fast ⚡️, has new solvers (eg. QR, tridiagonal), supports general linear Operators. github: github.com/google/lineax arXiv: arxiv.org/abs/2311.17283 1/n
- Tikhnov regularised trust-region methods (*cough* Levenberg-Marquardt) oddly use two different approximations to the objective function at each step. One regularised, one not. What if we just regularised both? 1/
- Hurrah! 🎉🥳 Very excited this is for this release, after many months of work! Lots of really cool things in Optimistix that I've wanted in optimisation software for a long time. I hope other researchers find as exciting as I do 😃Announcing a new #JAX and #Equinox nonlinear optimisation library: ⭐️ Optimistix ⭐️ (GitHub: github.com/patrick-kidger…) - Minimisation - Nonlinear least-squares - Root-finding - Fixed-points With blazing fast compile times and fast run times 🔥 1/7
