Jason Rader

@packquickly

PhD student @Oxford and very slow ultrarunner :)

Oxford/Malta

packquickly.com

Replies Reposts Media

Jason Rader
@packquickly
Oct 27, 2024
Adam depends on the gradient distribution during training, which, as far as I know, we don't understand well? Here, adapted from the Adam paper, v_t is the var estimate, G_t is the gradient r.v. and X_t is an error r.v. for distribution shift.
1
Jason Rader
@packquickly
May 29, 2024
Screw it, here's a new JAX implementation in Optimsitix: github.com/packquickly/sc…
Aaron Defazio
@aaron_defazio
May 28, 2024
Schedule-Free paper is up! arxiv.org/abs/2405.15682 Joint work with collaborators @alicey_ang @HarshMeh1a @konstmish @akhaledv2 @AshokCutkosky We have some strong small-scale experiments on Transformers, comparing to chinchilla-style cosine 10x reduction schedules.
2
Jason Rader
@packquickly
Nov 30, 2023
⭐ Lineax is now on arXiv! ⭐ If you’re doing linear solves or linear least-squares in JAX, give it a shot today! Lineax is fast ⚡️, has new solvers (eg. QR, tridiagonal), supports general linear Operators. github: github.com/google/lineax arXiv: arxiv.org/abs/2311.17283 1/n
GitHub - patrick-kidger/lineax: Linear solvers in JAX and Equinox. https://docs.kidger.site/lineax
From github.com
2
Jason Rader
@packquickly
Oct 11, 2023
Tikhnov regularised trust-region methods (*cough* Levenberg-Marquardt) oddly use two different approximations to the objective function at each step. One regularised, one not. What if we just regularised both? 1/
1
Jason Rader
@packquickly
Oct 6, 2023
Hurrah! 🎉🥳 Very excited this is for this release, after many months of work! Lots of really cool things in Optimistix that I've wanted in optimisation software for a long time. I hope other researchers find as exciting as I do 😃
Patrick Kidger
@PatrickKidger
Oct 6, 2023
Announcing a new #JAX and #Equinox nonlinear optimisation library: ⭐️ Optimistix ⭐️ (GitHub: github.com/patrick-kidger…) - Minimisation - Nonlinear least-squares - Root-finding - Fixed-points With blazing fast compile times and fast run times 🔥 1/7