1. X
  2. Stephen Roller
Log inSign up
Stephen Roller
Thinking Machines
6,624 posts
Image
user avatar
Stephen Roller
Thinking Machines
@stephenroller
MTS @thinkymachines. previously pre-training @googledeepmind, @character_ai, and @aiatmeta.
NYC
stephenroller.com
Joined February 2008
1,254
Following
5,970
Followers
RepliesRepliesMediaMedia

New to X?

Sign up now to get your own personalized timeline!

Create account

By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.

Terms·Privacy·Cookies·Accessibility·Ads Info·© 2026 X Corp.
Don't miss what's happening
People on X are the first to know.
Log inSign up
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Apr 29, 2024
    the surreal feeling of someone posting a paper internally, a new hire asking should we try it? and having to answer “yeah we call it flabberblanung internally. we’ve been using it as our default for about 8 months. here’s the code.” over and over and over and over.
    108K
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Jul 15, 2025
    We are moving incredibly fast. Come light up GPUs with us.
    user avatar
    Mira Murati
    @miramurati
    Jul 15, 2025
    Thinking Machines Lab exists to empower humanity through advancing collaborative general intelligence. We're building multimodal AI that works with how you naturally interact with the world - through conversation, through sight, through the messy way we collaborate. We're
    40K
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    May 3, 2022
    Super proud to have worked on this with @suchenzang, @NamanGoyal21 and many others.
    user avatar
    AI at Meta
    Meta
    @AIatMeta
    May 3, 2022
    Today Meta AI is sharing OPT-175B, the first 175-billion-parameter language model to be made available to the broader AI research community. OPT-175B can generate creative text on a vast range of topics. Learn more & request access: ai.facebook.com/blog/democrati…
    Image
    00:00
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Jan 30, 2023
    Happy to announce today is my first day @character_ai.
    58K
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Jun 14, 2024
    Some teams use sweeps, heuristics, or scaling laws to determine their training LR. At Character, we just have Noam Shazeer dial it to the right value.
    Image
    00:00
    170K
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Oct 12, 2022
    Replying to @srush_nlp
    I find people unfamiliar with scaling are shocked by this:
    Image
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    May 12, 2022
    Huge release from 🤗Transformers, including all the OPT models up to 30B parameters! You can even run OPT models in Colab now!
    user avatar
    Hugging Face
    @huggingface
    May 12, 2022
    Last week @metaai publicly released huge LMs, with up to ☄️30B parameters. Great win for Open-Source🎉 These checkpoints are now in 🤗transformers! But how to use such big checkpoints? Introducing Accelerate and ⚡️BIG MODEL INFERENCE⚡️ Load & USE the 30B model in colab (!)⬇️
    We load the checkpoint that is saved on disk and we dispatch it to the devices. At no point is the checkpoint fully loaded in RAM; only parts of it to be dispatched to each device.

We load it as float16 so that we may load more layers at a time on each device for a faster execution time.
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Apr 29, 2020
    Really excited to be sharing this with everyone today. Blog post below, paper here: arxiv.org/abs/2004.13637
    user avatar
    AI at Meta
    Meta
    @AIatMeta
    Apr 29, 2020
    Today we’re announcing that Facebook AI has built and open-sourced Blender, the largest-ever open-domain chatbot. It outperforms others in terms of engagement and also feels more human, according to human evaluators. ai.facebook.com/blog/state-of-…
    Image
    00:00
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Nov 13, 2020
    I'm looking for a PhD intern to join me summer 2021 at FAIR NY to work on Conversational AI. Interests include chit chat, task oriented dialogue, large-scale modeling, and evaluation. Apply at facebook.com/careers/jobs/1… and send me a heads up.
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Jan 10, 2020
    Mixout (arxiv.org/abs/1909.11299) is a cool way to regularize your large neural network. I quickly wrote an implementation in pytorch that works with (most) arbitrary nn.Modules: gist.github.com/stephenroller/…
    arXiv logo
    arxiv.org
    Mixout: Effective Regularization to Finetune Large-scale...
    In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus....
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Apr 28, 2017
    I have a PhD now. So now when people ask me "is it Stephen with a V or a PH?" I can reply, "It's Stephen with a PhD."
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    Nov 9, 2022
    Terrible day. A lot of great colleagues gone.
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    May 13, 2025
    I once trained hyperbolic (Poincaré) networks with Riemannian SGD and HogWild. Your optimization stack does not scare me.
    13K
  • user avatar
    Stephen Roller
    Thinking Machines
    @stephenroller
    May 4, 2022
    One of my biggest lessons from OPT is that engineering risk and research risk are multiplicative, and research risk can be easier to reduce (simplify to a known baseline).
Advertisement
Advertisement