Shuangyu Xie

PostDoc @ UCB, Ph.D. @ TAMU

photo.jpg
email google scholar linkedin x / twitter

I am a Postdoctoral researcher in the AUTOLab at UC Berkeley, working with Prof. Ken Goldberg. Prior to joining UC Berkeley, I obtained my Ph.D. and M.S. degree from Texas A&M University, advised by Prof. Dezhen Song. My research aims to make robots more capable and resilient in unstructured environments, spanning applications in manipulation, mobile manipulation and agricultural robotics.

Research Highlights

My recent work explores how Vision-Language Models and robotics can advance each other: real robot experience provides the grounded, embodied data that VLMs need, while VLMs bring the open-world semantic understanding that robots need to perceive, reason, and act. I pursue this along three threads:

  • Learning VLMs from robot experience. Robo2VLM (NeurIPS 2025, Spotlight) turns large-scale, in-the-wild robot manipulation datasets into grounded visual question-answering data, using the robot’s own sensing and kinematics to supervise VLM training and evaluation.
  • Evaluating VLMs for robotics. RoboVista (RSS 2026) is the first robot-at-work benchmark that evaluates VLMs across diverse real-world robot deployments—manipulation, driving, and surgical scenes—with a public leaderboard.
  • Understanding and curating robot data with VLMs. RoboSQ (ICRA 2026) is a semantic query engine that uses natural language to curate high-quality, task-aligned training data from large robot datasets, and RoboSegNet (CVPR 2026) fuses vision and kinematics to segment robot trajectories into meaningful sub-tasks, zero-shot.

I also build agentic harnesses that turn foundation models into deployable robot policies. GaP: Graph-as-Policy (CoRL 2026, Spotlight; with NVIDIA) represents a robot policy as an executable behavior graph: a team of LLM agents writes the graph from a plain-language task description, grounds it in an open robot-skills library, verifies it, and improves it in NVIDIA Isaac simulation before running it on real UR5e and Franka robots. Across grocery fulfillment, packing, popcorn making, cable insertion, and bimanual crate washing, GaP reaches 0.93–0.99 success under pose variation. Paper · Code

News

Oct 20, 2026 New blog post: Open Gemini 4 Argon Robot Harness, an open harness for Gemini-driven robot control, extended to two arms and tested on a zero-shot bimanual handover. Code.
Sep 29, 2026 GaP: Graph-as-Policy has been accepted to CoRL 2026 as a Spotlight! Paper · Code
Apr 20, 2026 RoboVista has been accepted to RSS 2026 — the leaderboard is now online, and the benchmark is available on Hugging Face!
Feb 26, 2026 RoboSegNet has been accepted to CVPR 2026!
Jan 29, 2026 RoboSQ has been accepted to ICRA 2026!
Sep 20, 2025 Robo2VLM has been accepted to NeurIPS 2025 as a Spotlight — the dataset has 29k+ downloads on Hugging Face!

Selected Publications

* indicates co-first author

  1. gap.jpg
    GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness for Variational Automation Tasks
    Kaiyuan Chen*, Shuangyu Xie*, Letian Fu, Justin Yu, William Pacini, Sandeep Bajamahal, Hudson Kim, Jaimyn Drake, Daehwa Kim, Haoru Xue, Jonathan Francis, Christian Juette, Peter Schaldenbrand, Muhammet Yunus Seker, Ruwan Wickramarachchi, Uksang Yoo, Guanzhi Wang, Adithyavairavan Murali, Balakumar Sundaralingam, S. Shankar Sastry, Spencer Huang, Yuke Zhu, Linxi Fan, and Ken Goldberg
    In Conference on Robot Learning (CoRL), 2026
    Spotlight
    arXiv Code Website
  2. robovista.png
    RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
    Shuangyu Xie*, Kaiyuan Chen*, Ziyang Chen, Simeon Adebola, Yixuan Huang, Zehan Ma, Tianshuang Qiu, Wentao Yuan, Dhruv Shah, Pannag R. Sanketi, and Ken Goldberg
    In Robotics: Science and Systems (RSS), 2026
    Website
  3. robosegnet.png
    Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams
    Kaiyuan Chen*, Shuangyu Xie*, Andrew Goldberg, and Ken Goldberg
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026
    Website
  4. robosq.png
    RoboSQ: Semantic Queries for Task-Aligned Robot Training Data
    Kaiyuan Chen*, Shuangyu Xie*, Kush Hari, Andrew Goldberg, Kavish Kondap, and Ken Goldberg
    In IEEE International Conference on Robotics and Automation (ICRA), 2026
    Website
  5. weed_cluster.png
    Energy Efficient Planning for Repetitive Heterogeneous Tasks in Precision Agriculture
    Shuangyu Xie, Ken Goldberg, and Dezhen Song
    In IEEE International Conference on Robotics and Automation (ICRA), 2025
  6. robo2vlm.png
    Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
    Kaiyuan Chen*, Shuangyu Xie*, Zehan Ma, and Ken Goldberg
    In Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025
    Spotlight
  7. botany-bot.png
    Botanybot : Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats
    Simeon Adebola, Chung Min Kim, Justin Kerr, Shuangyu Xie, Prithvi Akella, Jose Luis Susa Rincon, Eugen Solowjow, and Ken Goldberg
    In IROS, 2025
  8. capm.png
    Coupled Active Perception and Manipulation Planning for a Mobile Manipulator in Precision Agriculture Applications
    Shuangyu Xie, Chengsong Hu, Di Wang, Joe Johnson,  Muthukumar Bagavathiannan, and Dezhen Song
    International Conference on Robotics and Automation (ICRA), 2024
  9. flamer_robot_platform.png
    System and Algorithm for Robotic Weed Flaming Using a Quadruped Robot and a 6-Axis Manipulator
    Di Wang, Chengsong Hu, Shuangyu Xie, Joe Johnson, Hojun Ji, Yingtao Jiang, Muthukumar Bagavathiannan, and Dezhen Song
    International Conference on Intelligent Robots and Systems (IROS), 2024
  10. Dimenet.png
    Dynamic Intrinsic Parameter Rectification Network for Cameras with Optical Image Stabilization in Desktop Augmented Reality Applications
    Shuangyu Xie, Di Wang, Shu-Hao Yeh, Wei Yan, and Dezhen Song
    IEEE International Conference on Automation Science and Engineering (CASE), 2024
  11. pretouch.png
    A Pretouch Perception Algorithm for Object Material and Structure Mapping to Assist Grasp and Manipulation Using a DMDSM Sensor
    Fengzhi Guo, Shuangyu Xie,  Di Wang,  Cheng Fang,  Jun Zou, and  Dezhen Song
    International Conference on Intelligent Robots and Systems (IROS), 2023
  12. weed_spray.png
    Algorithm and System Development for Robotic Micro-Volume Herbicide Spray Towards Precision Weed Management
    Chengsong Hu, Shuangyu Xie, Dezhen Song, J. Alex Thomasson, Robert G. Hardin IV, and Muthukumar Bagavathiannan
    IEEE Robotics and Automation Letters, 2022
  13. weed_detection.png
    Toward Robotic Weed Control: Detection of Nutsedge Weed in Bermudagrass Turf Using Inaccurate and Insufficient Training Data
    Shuangyu Xie, Chengsong Hu, Muthukumar Bagavathiannan, and Dezhen Song
    IEEE Robotics and Automation Letters, 2021