Vincent Sitzmann

@vincesitzmann

Building AI that learns by interacting with the world. Associate Professor @ MIT, leading the Scene Representation Group (scenerepresentations.org).

Cambridge, Massachusetts

vincentsitzmann.com

Joined February 2016

344 Following

20.6K Followers

Replies Reposts Media

Pinned
Vincent Sitzmann
@vincesitzmann
Oct 20
Thrilled to introduce Mirror, our first "learning-from-experience" project! Assume you had a video generative model that, no matter your situation, could always generate video of what it would look like to solve a task. How can you put that plan into action? Clearly, you need to
Eric Ming Chen
@ericmchen1
Oct 20
If video models scale like LLMs, they'll soon generate a perfect video plan for any embodied task. Do we need anything else? Introducing Mirror, which learns to follow these plans just by practicing, and even crafts an iron pickaxe in Minecraft w/o rewards or action labels! (1/n)
00:00
11
Vincent Sitzmann
@vincesitzmann
23h
The old wisdom strikes again: go where the data is. Turns out there is a lot of video, so training a video planner is actually the *easy* path, albeit expensive. The truth of the matter is that the only way to get generality today is by training on diverse data at scale. Hence,
Hyunwoo Ryu
@RyuHyunwoooo
Oct 20
Replying to @RyuHyunwoooo
First: with expert data, do you need an action-conditioned world model? Not at all. Video planning is far better. In offline Minecraft, our video planner + IDM significantly outperforms Dreamer 4, the SOTA world-model agent: ~2x success on iron pickaxes, ~8x on diamonds. (2/15)
4
Vincent Sitzmann
@vincesitzmann
Oct 20
Posting about another really really exciting result tomorrow - my lab's first result in "learning from experience" / active learning / RL-like algorithms / whatever you might call it. Stay tuned!!
5
Vincent Sitzmann
@vincesitzmann
Oct 19
Very excited about this one: a completely new representation learning objective, with a proper theory of what it converges to: the representation of an image converges to features such that their dot product with another image's representation is their pointwise mutual
Alan Baade
@BaadeAlan
Oct 19
1/ The tokenizer is a new scaling axis! We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse. We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion w/ @drfeifei
00:00
5
Vincent Sitzmann
@vincesitzmann
Oct 16
Arthur is right: Europe has a recurring pattern of defeatism. The key to excellence is momentum: Arthur & Mistral have done a great job getting quite close to the frontier, at very small expenditure (relative to other big labs). Given more runway, they *will* get closer still.
Arthur Mensch
@arthurmensch
Oct 16
Should Europe leave frontier AI to the Americans and the Chinese? The idea has gained ground in Europe. It is a fundamental mistake. Build or surrender: for now, the choice is still ours. Full article below.
Article
Build or surrender
An idea has gained ground in Europe: we should leave frontier AI models to the Americans and the Chinese. It is advocated by serious people, whose reasoning has its logic. A frontier model costs
6