ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation

Official code release for ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation, by Jason Chen, I-Chun Arthur Liu, Gaurav S. Sukhatme, Daniel Seita (University of Southern California).

• Paper: arXiv:2509.19454
• Project page: https://ropaaug.github.io/

ROPA is an offline data augmentation method for vision-based imitation learning of bimanual manipulation. It fine-tunes Stable Diffusion 2.1 with ControlNet, conditioned on robot skeleton poses, to synthesize third-person observations of novel robot poses. The synthesized frames are paired with joint-space action labels enforced through constrained optimization. Policies trained on the augmented dataset reach higher success rates than baselines on 5 simulated and 3 real-world tasks.

1. Pipeline overview
2. Repository layout
3. Installation
4. What you need to bring yourself
5. Training
6. Validation and Test
7. Tasks supported
8. Path handling
9. Acknowledgements
10. Citation
11. License

RLBench demos ──► perturbed-joint JSON ──► skeleton-pose renders ──► ControlNet training set ──►
ControlNet fine-tune ──► augmented RGB + joint actions ──► ACT training ──► held-out eval

See docs/PIPELINE.md for the per-stage commands with example paths.

ROPA_FINAL/
├── README.md                        this file
├── docs/
│   ├── INSTALL.md                   per-component environment setup
│   ├── PIPELINE.md                  step-by-step run guide
│   └── PATHS.md                     hardcoded paths to edit per script
├── env.example                      ROPA_DATA_ROOT and friends — copy to .env or export
├── requirements.txt                 top-level Python deps (per-component reqs live in each dir)
├── controlnet/                      SD 2.1 + ControlNet training and inference
├── render/                          skeleton-pose renderer (PyRender + URDF)
├── preprocess/                      dataset construction, joint perturbation, k=8 replacement
└── peract_bimanual/                 ACT training & evaluation (PerAct2-derived)
    └── scripts_train_eval/DCODA/    per-task ROPA training and eval scripts

ROPA uses four separate conda environments because component dependencies conflict (SD 2.1 needs a specific torch + pytorch-lightning pair; PerAct2 has its own torch pin; renderer uses Hydra/PyRender):

Full setup details, troubleshooting, and OOM tips: docs/INSTALL.md.

git clone <this-repo> ROPA_FINAL && cd ROPA_FINAL
cp env.example .env
# edit .env: set ROPA_DATA_ROOT to wherever your RLBench demos and augmented datasets live
source .env

The release contains source code only. You must separately obtain:

Run the 7 stages in order; full command examples are in docs/PIPELINE.md. All paths default under ${ROPA_DATA_ROOT}.

1. Generate demos — python RLBench/tools/dataset_generator_bimanual.py --tasks=coordinated_lift_ball --episodes_per_task=100 --image-size=128x128 (run inside xvfb-run).
2. Sample perturbed joints — python preprocess/add_demonstrations_rlbench_json_file_withconstraintoptimization.py
3. Render skeleton poses — python render/render_data.py (with use_custom_joints=False)
4. Build ControlNet training set — python preprocess/controlNet_data_preprocessing_bimanual_genima.py 5a. Fine-tune ControlNet — python controlnet/train_controlnet_ropa.py. All knobs (init checkpoint, resume path, batch size, max epochs, output dir) are constants at the top of that file — edit them there. The single-GPU/fp32 tutorial_train_sd21.py is kept for reference only. 5b. Synthesize the augmented dataset — python controlnet/add_demonstrations_rlbench_GEMINA_step1.py then step2.py
5. Train ACT — bash peract_bimanual/scripts_train_eval/DCODA/train_act_coordinated_lift_ball.sh (vary framework.start_seed for multi-seed runs)

Validation rolls out every saved checkpoint on a held-out demo split; test rolls out only the val-best checkpoint on a disjoint split. Generate the two held-out splits the same way as the training demos and place them under ${ROPA_DATA_ROOT}/demos_val/ and ${ROPA_DATA_ROOT}/demos_test/.

# Validation: sweep every checkpoint, picks best
python eval.py method=ACT_BC_LANG rlbench.task_name=<EXP_NAME> \
    rlbench.demo_path=${ROPA_DATA_ROOT}/demos_val \
    framework.eval_episodes=25 framework.eval_type=missing framework.eval_envs=4

# Test: run the best step from the validation sweep on demos_test
python eval.py method=ACT_BC_LANG rlbench.task_name=<EXP_NAME> \
    rlbench.demo_path=${ROPA_DATA_ROOT}/demos_test \
    framework.eval_episodes=25 framework.eval_type=<best_step_int>

The validation sweep writes per-step success rates to <exp_dir>/eval_data.csv; pick the argmax row's step and pass it as eval_type=<step> for the test pass. Per-task ready-made eval scripts: peract_bimanual/scripts_train_eval/DCODA/.

Five RLBench bimanual tasks from PerAct2:

• coordinated_lift_ball (CLB)
• coordinated_lift_tray (CLT) — task variants easy5..easy11
• coordinated_push_box (CPB)
• bimanual_straighten_rope (BSR)
• coordinated_put_item_in_drawer (CPID)

Per-task train and eval scripts live in peract_bimanual/scripts_train_eval/DCODA/.

All shell scripts use ${ROPA_DATA_ROOT:-/data} so they work out-of-the-box if your data is at /data, or can be redirected via the environment variable. The renderer config (render/cfgs/render.yaml) reads ${oc.env:ROPA_DATA_ROOT,/data}.

Python entrypoints still contain literal /data/... paths in their __main__ blocks because they're typically run as one-off experiments. docs/PATHS.md lists every script and the line numbers you need to update before running.

ROPA builds directly on prior work — please cite each as appropriate when using this code:

• PerAct2 (bimanual RLBench fork, ACT agent, YARR/ARM training stack) — github.com/markusgrotz/peract_bimanual
• ControlNet (skeleton-conditioned diffusion) — github.com/lllyasviel/ControlNet — Apache 2.0
• Stable Diffusion 2.1 — stabilityai/stable-diffusion-2-1 — CreativeML OpenRAIL++
• GENIMA (skeleton-pose renderer) — github.com/MohitShridhar/genima
• RLBench / PyRep — github.com/stepjam/RLBench, github.com/stepjam/PyRep
• ACT (Action Chunking Transformer) — github.com/tonyzhaozh/act
• D-CODA (related augmentation work; informs our val/test split design) — github.com/dcodaaug/dcoda
• VISTA — see paper for citation

See the paper for full attributions.

@article{chen2025ropa,
  title   = {ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation},
  author  = {Chen, Jason and Liu, I-Chun Arthur and Sukhatme, Gaurav S. and Seita, Daniel},
  journal = {arXiv preprint arXiv:2509.19454},
  year    = {2025}
}

MIT — see LICENSE. Bundled third-party code (CLIP, ControlNet, latent-diffusion, mmcv, PerAct2/ARM/YARR fragments) retains its original license.