Seungjae (Jay) Lee

이승재

I am a Ph.D. student in the Department of Computer Science at UMD, where I am fortunate to be co-advised by Dr. Furong Huang and Dr. Jia-Bin Huang. I am also a Research Intern at NVIDIA GEAR Lab, where I have the pleasure of working with Dr. Jim Fan, Dr. Danfei Xu, and Dr. Ruijie Zheng.

Prior to joining UMD, I earned my Master's degree in the Department of Aerospace Engineering at SNU, where I was fortunate to be advised by Dr. H. Jin Kim. I also had the wonderful opportunity to work on robot learning with Dr. Lerrel Pinto of NYU.

Before that, I received Bachelor's degrees in Mechanical and Aerospace Engineering at SNU.

News

Organizing the Scaling H2R workshop at CoRL 2026
DynaFLIP is accepted to NeurIPS 2026
μ₀ and TAPESTRY are accepted to CoRL 2026
Started internship at NVIDIA GEAR team
TraceGen is accepted to CVPR 2026
MomaGraph is accepted to ICLR 2026

Education & Experiences

Education

Ph.D. in Computer Science
University of Maryland

Advised by Professor Furong Huang and Professor Jia-Bin Huang.

Aug 2024 - Present | College Park, MD

M.S. in Aerospace Engineering
Seoul National University

Advised by Professor H. Jin Kim.

Mar 2021 - Feb 2024 | Seoul, Korea

B.S. in Mechanical & Aerospace Engineering
Seoul National University

Mar 2015 - Feb 2021 | Seoul, Korea

Experiences

NVIDIA

GEAR Team Research Intern

May 2026 - | Santa Clara, CA

Toyota Research Institute

Large Behavior Model Team Research Intern

May 2025 - Aug 2025 | Cambridge, MA

New York University

Research collaborator (Remote), advised by Professor Lerrel Pinto.

Jul 2023 - Jun 2024 | Seoul, Korea (Remote)

Samsung Electronics

Deep Learning Algorithm Team Intern

Jul 2020 - Sep 2020 | Gyunggi-do, Korea

Research

Projects I led or co-led

My research aims to scale robot learning beyond expensive robot data: drawing on human videos and vision-language models, and making perception and action generation more efficient.

Current research interests

Learning from Human Videos
- Dex-One2ManyPreprint ’26
- HumanEgoPreprint ’26
- μ₀CoRL ’26
- TraceGenCVPR ’26
VLMs for Embodied Agents
- Fly-by-CodePreprint ’26
- Beyond Episodic EvaluationIROS ’26
- MomaGraphICLR ’26
- Imagine, Verify, ExecuteCoRL ’25
Efficient Action Generation
- TAPESTRYCoRL ’26
- DASIPNeurIPS ’25
- Robot Utility ModelsICRA ’25
- VQ-BeTICML ’24
Visual & Tactile Representations
- What to Attend, What to KeepPreprint ’26
- DynaFLIPNeurIPS ’26
- SNeRLICML ’23

Other areas I still love

Web AI Agent Security
- Why Are Web AI Agents More Vulnerable?ICLRw ’25
Exploration for Reinforcement Learning
- CQMNeurIPS ’23
- Diversify & ConquerNeurIPS ’23
- OUTPACEICLR ’23
- DHRLNeurIPS ’22

Publications

2026

Preprint2026

Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback
Jaewoo Lee*, Jeongyeon Seo*, Sihyun Cho*, Gyeongrak Choe, Yutong Wang, Bavin Saravanan, Jia-Bin Huang, Furong Huang, Sebastian Scherer, Guanya Shi, H. Jin Kim, Seungjae Lee†, Dongjae Lee† (*equal contribution, †equal advising)
project website

Preprint2026

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Jusuk Lee*, Sungha Kim*, Yeonsoo Park*, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang†, Seungjae Lee† (*equal contribution, †equal advising)
project website / arXiv

Preprint2026

What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory
Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene, Shanthosh Raaj Mohanram Mageswari, Soroush Etemad, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos
project website / arXiv
IROS 2026 Workshop Touch2Action (🏆 Best Paper Finalist)

Preprint2026

HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
Zhi Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, Yiannis Aloimonos
Under review
project website / arXiv / github
IROS 2026 Workshop WORLDS (🏆 Best Paper Award - Winner)

NeurIPS2026

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Jusuk Lee, Seungjae Lee, Jonghun Shin, Hoseong Jung, Sungha Kim, Daesol Cho, H. Jin Kim, Jia-Bin Huang†, Furong Huang† (†equal advising)
project website / arXiv

CoRL2026

μ₀: A Scalable 3D Interaction-Trace World Model
Seungjae Lee*, Yoonkyo Jung*, Jusuk Lee, Jonghun Shin, Amir Hossein Shahidzadeh, Yao-Chih Lee, H. Jin Kim, Jia-Bin Huang†, Furong Huang† (*equal contribution, †equal advising)
project website / arXiv
IROS 2026 Workshop RoBoWoMo (🏆 Best Paper Award - 2nd Place)

CoRL2026

TAPESTRY: A Single Backbone for Video Generation and Robot Control
Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah
project website

IROS2026

Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering
Zikui Cai*, Kaushal Janga*, Tan Dat Dao*, Seungjae Lee*, Shivin Dass, Mingyo Seo, Kaiyu Yue, Mintong Kang, Nandhu Pillai, Monte Hoover, Aadi Palnitkar, Ruchit Rawal, Ruijie Zheng, Bo Li, Yuke Zhu, Roberto Martín-Martín, Tom Goldstein, Furong Huang (*equal contribution)
arXiv

CVPR2026

TraceGen: World Modeling in 3D Trace-Space Enables Learning from Cross-Embodiment Videos
Seungjae Lee*, Yoonkyo Jung*, Inkook Chun*, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang (*equal contribution)
project website / arXiv / github

ICLR2026 🏆 Oral

MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning
Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath
project website / arXiv

2025

NeurIPS2025

Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Inkook Chun, Seungjae Lee, Michael Samuel Albergo, Saining Xie, Eric Vanden-Eijnden
arXiv / github

CoRL2025

Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
Seungjae Lee*, Daniel Ekpo*, Haowen Liu, Furong Huang†, Abhinav Shrivastava†, Jia-Bin Huang† (*equal contribution, †equal advising)
project website / arXiv

ICLRw2025

Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
Jeffrey Yang Fan Chiang*, Seungjae Lee*, Jia-Bin Huang, Furong Huang, Yizheng Chen (*equal contribution)
project website / arXiv

ICRA2025

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
project website/ paper/ github
CoRL 2024 Workshop on Language and Robot Learning (🏆 Oral)

2024

ICML2024 🏆 Spotlight, Top 3.5%

Behavior Generation with Latent Actions
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
project website / arXiv / github / 🤗 Lerobot Library
RSS 2024 Workshop SemRob (🏆 Oral spotlights) ICML 2024 Workshop MFM-EAI (🏆️ Outstanding Paper Award - Winner)

2023

NeurIPS2023

CQM: Curriculum Reinforcement Learning with a Quantized World Model
Seungjae Lee, Daesol Cho, Jonghae Park, H Jin Kim
arXiv

NeurIPS2023

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Daesol Cho, Seungjae Lee, H Jin Kim
arXiv

ICML2023

SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning
Dongseok Shim*, Seungjae Lee*, H Jin Kim (*equal contribution)
arXiv / github

ICLR2023 🏆 Spotlight, Top 5.65%

Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation
Daesol Cho*, Seungjae Lee*, H Jin Kim (*equal contribution)
arXiv / github

IJCAS2023

Deep End-to-End Imitation Learning for Missile Guidance with Infrared Images
Seungjae Lee, Jongho Shin, Hyeong-Geun Kim, Daesol Cho, H. Jin Kim

2022

NeurIPS2022 🏆 Oral, Top 1.76%

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Lee, Jigang Kim, Inkyu Jang, H. Jin Kim
arXiv / github

2021

IROS2021

Robust and Recursively Feasible Real-Time Trajectory Planning in Unknown Environments
Inkyu Jang, Dongjae Lee, Seungjae Lee, H Jin Kim
arXiv

Code

huggingface/lerobot 28.4k Making AI for robotics more accessible with end-to-end learning vq_bet_official 221 Behavior Generation with Latent Actions — ICML 2024 Spotlight TraceGen 64 World modeling in 3D trace-space from cross-embodiment videos — CVPR 2026 Yoonkyo/mu0 71 A scalable 3D interaction-trace world model — CoRL 2026 dhrl_official 34 Graph-based long-horizon, sparse hierarchical RL — NeurIPS 2022 Oral snerl_official 32 Semantic-aware neural radiance fields for RL — ICML 2023 outpace_official 12 Curriculum goal generation for outcome-directed RL — ICLR 2023 Spotlight imagine_verify_execute 3 Memory-guided agentic exploration with VLMs — CoRL 2025

Awards and Achievements

[Awards] Graduated Summa Cum Laude, Seoul National University (1st in Department of Aerospace Engineering)
[Scholarship] Daishin Songchon Foundation (Aug 2024 - Present)
[Scholarship] Hyundai Motor Chung Mong-Koo Foundation (Aug 2021 - Jul2023)
[Scholarship] OK Bae & Jung Scholarship Foundation (Mar 2020 - Dec 2020)

Academic Services

Organizer, Scaling H2R: Scaling Laws and Diversity in Human-to-Robot, CoRL 2026
Program Committee, RSS 2024 SemRob Workshop
Conference reviewer for ICML'22 '24 '25
Conference reviewer for IROS'23
Conference reviewer for NeurIPS'23 '25 '26
Conference reviewer for ICLR'24 '25 '26
Conference reviewer for ICRA'24 '25
Conference reviewer for AAAI'25
Conference reviewer for CoRL'25 '26
Conference reviewer for RSS'25
Conference reviewer for CVPR'26

Teaching

[TA] Introduction to Datascience (CMSC320), University of Maryland (2026 Spring)
[TA] AI Planning (CMSC722), University of Maryland (2024 Fall)
[TA] Hyundai Motor Company AI Boosting Camp, Seoul National University (2023 Summer)
[TA] Principles of Flight Vehicle Control (M2795.002600), Seoul National University (2021 Spring)
[TA] Basic Physics 1 (034.005), Basic Physics 2 (034.006), Seoul National University (2016 Spring, 2019 Spring, 2019 Fall)

News

Education & Experiences

Education

Ph.D. in Computer Science University of Maryland

M.S. in Aerospace Engineering Seoul National University

B.S. in Mechanical & Aerospace Engineering Seoul National University

Experiences

NVIDIA

Toyota Research Institute

New York University

Samsung Electronics

Current research interests

Learning from Human Videos

VLMs for Embodied Agents

Efficient Action Generation

Visual & Tactile Representations

Web AI Agent Security

Exploration for Reinforcement Learning

Publications

Code

Awards and Achievements

Academic Services

Teaching

Ph.D. in Computer Science
University of Maryland

M.S. in Aerospace Engineering
Seoul National University

B.S. in Mechanical & Aerospace Engineering
Seoul National University