In-context learning for robotics is here.
- Long-horizon tasks over 10 minutes long
- Never seen during pre-training
- Prompted with one video, no fine-tuning
We are building intelligence from the foundations up, not from the top down.
Introducing S1, our new foundation model that learns from one example.
It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning.
Watch S1 operate in real-time via in-context learning:
You don't have to post-train ChatGPT on every user. If you did, it would never have taken off. Yet this is exactly how robotics works today
For robots to take off, they need to learn in-context.
Great article from @chris_j_paxton on this new paradigm and S1's place in it:
In-Context Learning Results Hint at a “GPT Moment” for Robotics
For general-purpose robots to be useful and economical, you need to be able to teach them new skills on the fly. And now we can see how this will work, with in-context long horizon video demonstrations shown by
Exciting day for NVIDIA and @huggingface.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI.
Thank you
One surprising aspect of S1’s in-context learning (ICL) is where it shines most: super long-horizon tasks (10+ min) and scenarios outside the pretraining distribution.
For short, simple pick-and-place tasks (5–20 sec), most frontier models can already perform well either
Introducing S1, our new foundation model that learns from one example.
It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning.
Watch S1 operate in real-time via in-context learning:
Feels nice when folks actually read a long blog carefully. Makes countless hours spent with the team obsessing over every single word totally worth it!
This is something I learned the hard way over the years from my dear advisor -- Alyosha Efros. Those who know him, know. Haha.