Samuel L Smith (@SamuelMLSmith) / X

Samuel L Smith

286 posts

Samuel L Smith

@SamuelMLSmith

Member of Technical Staff at OpenAI. Formerly Staff Research Scientist at Google DeepMind. Ex-Physicist.

Joined January 2021

Pinned
Samuel L Smith
@SamuelMLSmith
Oct 13, 2025
The Training team @OpenAI is hiring researchers in London 🚀 Our twin missions are to train better LLMs, and serve them more cheaply Get in touch if you are excited to collaborate on architecture design, reliable scaling, and faster optimization
91K
Samuel L Smith
@SamuelMLSmith
Mar 4, 2025
First day at OpenAI London 😁!
48K
Samuel L Smith
@SamuelMLSmith
Feb 12, 2021
Proud to be a part of NFNets, a new ImageNet SOTA: - does not use BatchNorm, LayerNorm, GroupNorm, anyNorm! - 86.5% top-1 w/o extra data - 89.2% top-1 w/ pre-training - 8.7x faster than EffNet-B7 to same test accuracy arxiv.org/abs/2102.06171 code: dpmd.ai/nfnets 1/4
Samuel L Smith
@SamuelMLSmith
Apr 9, 2024
Announcing RecurrentGemma! github.com/google-deepmin… - A 2B model with open weights based on Griffin - Replaces transformer with mix of gated linear recurrences and local attention - Competitive with Gemma-2B on downstream evals - Higher throughput when sampling long sequences
178K
Samuel L Smith
@SamuelMLSmith
Sep 24, 2024
Replying to @srush_nlp
I think there is a well established answer. We initialize and optimize deep networks in such a way that the model explores simple functions before complex ones. Although overfit functions exist in weight space they are usually much harder to find.
17K
Samuel L Smith
@SamuelMLSmith
Sep 25, 2024
I wish "ResNets" and "Transformers" were called "ResConvs" and "ResAttn". ResNet should be an umbrella term for any deep network with a repeating pattern of skip connections and residual branches.
19K
Samuel L Smith
@SamuelMLSmith
Oct 18, 2024
I ambushed a theory workshop with a tutorial on scaling LLMs: youtu.be/GfAT2zkB6-U?si… Covers transformers, a simple model of how TPUs work, how to train models that don't fit on a single device, scaling plots and how training/inference differ
20K
Samuel L Smith
@SamuelMLSmith
Mar 1, 2024
Incredibly excited to announce Hawk and Griffin (arxiv.org/abs/2402.19427), two recurrent language models with 1) finite sized state + fast inference 2) efficient training on device 3) excellent performance:
28K
Samuel L Smith
@SamuelMLSmith
Jan 29, 2021
The Stochastic Gradient Descent we use in practice, SGD with Random Shuffling, is not a Stochastic Differential Equation when the learning rate is small. Instead, it follows the path of gradient flow on a regularized loss: arxiv.org/abs/2101.12176 (Mea Culpa at ICLR 2021)
Samuel L Smith
@SamuelMLSmith
Oct 26, 2023
ConvNets Match Vision Transformers at Scale: arxiv.org/abs/2310.16764 We scale NFNet pre-training on JFT-4B from 0.4 to 110k TPU-v4 core hours. After fine-tuning, our largest model achieves 90.4% ImageNet Top-1, competitive with ViTs pre-trained for similar compute budgets. 1/3
96K
Samuel L Smith
@SamuelMLSmith
Jun 12, 2024
RecurrentGemma-9B is out! kaggle.com/models/google/… huggingface.co/google/recurre… - Uses Griffin architecture, combining linear recurrence with local attention - Downstream evals comparable to Mistral and Gemma - Faster inference, especially for long sequences or large batch sizes 1/n
33K
Samuel L Smith
@SamuelMLSmith
Oct 4, 2024
Replying to @fchollet
The lesson we took from working on Griffin (arxiv.org/pdf/2402.19427) is that current model performance is bottlenecked by the channel mixing component (ie the MLP), not the sequence mixing component (ie Attention vs recurrence)
8K
Samuel L Smith
@SamuelMLSmith
May 4, 2021
Want to discuss how SGD implicitly regularises NN training, or how to train ResNets without BatchNorm. Come join our two posters @iclr_conf today (Tuesday) at 5-7pm UK time: SGD/Implicit Regularization: iclr.cc/virtual/2021/p… Norm-Free ResNets: iclr.cc/virtual/2021/p…
Samuel L Smith
@SamuelMLSmith
Sep 24, 2024
Replying to @srush_nlp
You could give them a take home 😀: form a dataset of 5 MNIST 1's and 0's. Train a small MLP to catastrophically overfit (100% train accuracy, <55% test accuracy). Spoiler: this is possible if you initialize the first weight matrix too large and use a v small learning rate.
2.2K