[Project page] [Paper] [Hardware Guide]

Xiaomeng Xu¹, Dominik Bauer², Shuran Song^1,2

¹Stanford University, ²Columbia University,

pip install -r requirements.txt

Single-GPU training:

python train.py --config-name=train_diffusion_transformer_snake_workspace task.dataset_path=dataset.zarr.zip

Multi-GPU training:

accelerate launch --num_processes <ngpus> train.py --config-name=train_diffusion_transformer_snake_workspace task.dataset_path=dataset.zarr.zip

Downloading sweeping dataset (processed):

wget https://real.stanford.edu/robopanoptes/data/zarr_datasets/sweep.zarr.zip

Multi-GPU training:

accelerate launch --num_processes <ngpus> train.py --config-name=train_diffusion_transformer_snake_workspace task.dataset_path=sweep.zarr.zip

The unboxing and stowing datasets are also available at https://real.stanford.edu/robopanoptes/data/zarr_datasets/.

3D print the parts and assemble the robot according to our Hardware Guide.

pip install dynamixel_sdk

Install the dynamixel_wizard. By default, each motor has the ID 1. In order for multiple dynamixels to be controlled by the same U2D2 controller board, each dynamixel must have a unique ID. This process must be done one motor at a time. Connect each motor, starting from the base motor, and assign them in increasing order (0~9).

• Connect a single motor to the controller and connect the controller to the computer
• Open the dynamixel wizard
• Click scan (top left corner), this should detect the dynamixel. Connect to the motor
• Look for the ID address and change the ID to the appropriate number
• Repeat for each motor

Find dynamixel control box port map:

ls /dev/serial/by-id

Look for the path that starts with usb-FTDI_USB__-__Serial_Converter, set the port in eval_real.py.

ls /dev/v4l/by-path

Look for the paths that look like /dev/v4l/by-path/pci-0000:00:14.0-usb-0:2:1.0-video-index0, find the corresponding id for each camera. Set the device_ids in eval_real.py.

Download pre-trained checkpoint.

wget https://real/stanford.edu/robopanoptes/data/pretrained_models/sweep.ckpt

Launch eval script.

python eval_real.py

ls /dev/serial/by-id

Look for the paths that start with usb-FTDI_USB__-__Serial_Converter, find the corresponding client_port and leader_port, and replace the args in teleop/run_env.py.

Set the device_ids in teleop/run_env.py.

python teleop/run_env.py

python process_data.py

@article{xu2025robopanoptes,
	title={RoboPanoptes: The All-seeing Robot with Whole-body Dexterity},
	author={Xu, Xiaomeng and Bauer, Dominik and Song, Shuran},
	journal={arXiv preprint arXiv:2501.05420},
	year={2025}
}	

This repository is released under the MIT license. See LICENSE for more details.

• Our diffusion policy implementation is adapted from UMI.
• Our teleoperation code and guide are adapted from GELLO.