Peter Tong

News

• 2026-01: New blog post: Lessons from Two Years of TPU Training in Academia — sharing our hard-earned TPU debugging wisdom!
• 2025-06: Our papers Web-SSL and MetaMorph were accepted at ICCV 2025!
• 2025-05: I am re-joining FAIR as a research scientist intern with the amazing Koustuv Sinha!
• 2024-09: Our paper RLVLM was accepted at NeurIPS2024, and Cambrian was accepted at NeurIPS 2024 as an Oral Paper!
• 2024-05: I joined FAIR, Meta for summer internship with Dr. Zhuang Liu, yayyyyy!
• 2024-04: I received OpenAI Superalignment Fellowship! Thank you OpenAI!!! Looking forward to the cool works.
• 2024-04: Our paper Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs was accepted at CVPR 2024 as an Oral Paper!
• 2024-01: Our paper Image Clustering in the Age of Pretrained Models was accepted at ICLR 2024!
• 2023-12: Our papers were accepted at CPAL 2024!
• 2023-09: I am helping organizing the QVCV workshop in ICCV. See you all in Paris!
• 2023-09: Our papers MultiMon and CRATE(whitebox-transformer) were accepted at NeurIPS 2023!
• 2023-07: Our paper Manifold Linearizing and Clustering was accepted at ICCV 2023!
• 2023-05: I graduated from UC Berkeley with triple degree Applied Math (Honor), Statistics (Honor) and Computer Science (No honor, because I didn't want to take 16b and 61c too early, but I published quite some interesting work so yay)!!!
• 2023-04: I will be a CS PhD student in NYU Courant advised by Professor Yann LeCun and Professor Saining Xie. Looking forward to working with Yann and Saining in New York!
• 2023-01: Our paper incremental-CTRL was accepted at ICLR 2023!

Blogs

• Lessons from Two Years of TPU Training in Academia

  Shengbang Tong, Boyang Zheng · Jan 2026

  Hard-earned lessons from thousands of hours debugging TPUs (v3-v6e): static shapes, TorchXLA pitfalls, SPMD sharding, and data storage strategies for academic ML research.

Publications

Representation Autoencoders (RAE)

We show pretrained representation encoders (DINO, SigLIP, MAE) paired with trained decoders serve as better latent spaces for diffusion than VAEs — enabling faster convergence, richer semantics, and higher generation quality. We thoroughly develop the method in the ImageNet setting, then demonstrate it scales effectively to large-scale text-to-image generation.

Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders

Shengbang Tong*, Boyang Zheng*, Ziteng Wang*, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining Xie

Technical Report, Jan 2026

Diffusion Transformers with Representation Autoencoders

Boyang Zheng, Nanye Ma, Shengbang Tong, Saining Xie

ICLR 2026

Scaling Language-Free Visual Representation Learning

David Fan*, Shengbang Tong*, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar†, Saining Xie†

ICCV 2025 Highlight

We introduce Visual SSL 2.0: Scaling up models, data to billion scale and adding VQA to the evaluation suite. Vision-only models scale with model size and data size, eventually catching up/surpassing CLIP models.

MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu

ICCV 2025

Visual understanding and visual generation are mutually beneficial in unified models! But visual understanding data is much more effective than visual generation. Capabilities in LLM can also transfer to unified models such as implicit reasoning!

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Shengbang Tong*, Ellis Brown*, Penghao Wu*, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, Saining Xie

NeurIPS 2024 Oral

We provide a vision-centric exploration or cookbook in MLLMs, systematically studying visual representation, vision-language connector, instruction tuning data, training recipe and evaluation protocols. We propose new vision-centric benchmarks, spatial-aware connector, data collection and curation of instruction data, and release very competitive 8B, 13B and 34B models on par with GPT-4V and Gemini.

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie

CVPR 2024 Oral

Is vision good enough for language? Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. We identify 'CLIP-blind pairs' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark.

Mass-Producing Failures of Multimodal Systems with Language Models

Shengbang Tong*, Erik Jones*, Jacob Steinhardt

NeurIPS 2023

Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MULTIMON, a system that automatically identifies systematic failures.