[🚀 Quick Start] [🌟 Pre-train] [🤖 Post-train (Robotwin)] [π0.5 Baseline] [🕹 RoboTwin Eval] [🐛 Troubleshooting]
2026/08: Release the runnable PI0.5 RoboTwin baseline with isolated training and evaluation entries; the validated checkpoint is available on ModelScope.2026/08: Release 3D Grounding inference script (infer_grounding3d.py) — supports real-world RGB-D input and HDF5 data for open-vocabulary 3D object grounding.2026/08: Upload pretrained PoseVLA Stage-1 checkpoint to Hugging Face.2026/06: Initial release of PoseVLA: supports Omni3D / Omni6D / BOP / GraspClutter6D for 3D tasks and Agibot / InternData-A1 for robot actions.
| Method | Easy | Hard |
|---|---|---|
| π0 | 67.00 | 65.12 |
| π0.5 | 79.48 | 76.16 |
| PaliGemma_expert | 35.40 | 33.36 |
| Pose-VLA (In Paper) | 79.91 | 79.10 |
| PoseVLA (In Repo) | 89.40 | 88.60 |
PoseVLA is split into two training stages, each with its own document:
| Stage | Entry script | Config | Doc |
|---|---|---|---|
| Pre-train (joint VLM + Action on large-scale data) | train_pretrain.py | config/base.yaml | docs/PRETRAIN.md |
| Robotwin data conversion (raw → HDF5 + normalization) | utils/process_data_all.py | config/dataset/robotwin.yaml | docs/ROBOTWIN_DATA.md |
| Post-train / Fine-tune (Robotwin and other downstream robots) | train_posttrain.py | config/base_posttrain.yaml | docs/POSTTRAIN.md |
| PI0.5 RoboTwin baseline (14-D qpos train + eval) | pi05_baseline/train.py | pi05_baseline/config/train.yaml | pi05_baseline/README.md · Checkpoint |
| 3D Grounding Inference (real-world RGB-D / HDF5) | infer_grounding3d.py | config/base.yaml | below |
| RoboTwin simulation eval | robotwin/PoseVLA/eval_policy.py | robotwin/PoseVLA/deploy_policy.yml | robotwin/PoseVLA/README.md |
The project supports three orthogonal training modes that can be freely combined:
- VLM training — learn 3D object detection via Next-Token Prediction (NTP) on Omni3D, Omni6D, BOP, GraspClutter6D, …
- Action training — learn robot actions via Flow Matching on HDF5 / LeRobot-format data (Agibot, InternData-A1, Robotwin, …).
- Co-Training — VLM and Action data are interleaved within the same optimization step.
📁 Project Structure (click to expand)
PoseVLA/
├── train_pretrain.py # Pre-train entry (hydra + accelerate + deepspeed)
├── train_posttrain.py # Post-train / fine-tune entry (Robotwin etc.)
├── eval_detection.py # Evaluation / mAP entry (Omni3D and other 3D tasks)
├── infer_grounding3d.py # 3D grounding inference (real-world RGB-D / HDF5)
│
├── posevla/ # PoseVLA model implementation (π0 / π0.5 based)
│ ├── configuration_posevla.py # PoseVLAConfig
│ ├── modeling_posevla.py # PoseVLAPolicy (PaliGemma + Action Expert + Flow Matching)
│ ├── paligemma_with_expert.py # PaliGemma + Expert dual-stream architecture
│ ├── patch_embed.py # Vision patch embedding (SigLIP + prior fusion)
│ ├── convert_jax_model_to_pytorch.py # JAX → PyTorch weight converter
│ └── _lerobot_compat.py # LeRobot compatibility layer
│
├── data/
│ ├── factory.py # Unified VLM / Action DataLoader factory
│ ├── collators.py # DataCollators (action / detection)
│ ├── ds_raw/ # Raw dataset readers
│ │ ├── agibot.py / interndata_a1.py / robotwin.py
│ │ ├── droid.py / rdt.py / umi.py / xtrainer.py
│ │ └── mix.py # Multi-source mixed dataset reader
│ └── ds_train/ # Training Datasets
│ ├── detection/ # 3D detection datasets (VLM)
│ │ ├── dataset_bop.py / dataset_omni3d.py / dataset_omni6d.py
│ │ ├── dataset_clutter.py
│ │ └── graspclutter6dAPI.py
│ └── robot/ # Robot action datasets (VLA)
│ ├── dataset_hdf5_action.py # HDF5 action dataset
│ ├── dataset_lerobot.py # LeRobot format dataset
│ └── dataset_agibot.py # Agibot dataset
│
├── utils/ # Shared utilities
│ ├── process_data_all.py # Robotwin raw → HDF5 conversion
│ ├── mapping_token.py # Text ↔ 3D scene encoding / decoding utilities
│ ├── transform_utils.py # SE(3) / pose math helpers
│ ├── image_corrupt.py # Image augmentation
│ ├── vis.py # Visualization helpers
│ └── logger.py # WandB / training-state logging
│
├── config/ # Hydra configs
│ ├── base.yaml # Pre-train main config
│ ├── base_posttrain.yaml # Post-train (Robotwin) main config
│ ├── zero0.json / zero2.json / zero3_offload.json # DeepSpeed configs
│ ├── dataset/ # Action training dataset configs (hdf5, lerobot, robotwin)
│ ├── dataset_bop/ # BOP series
│ ├── dataset_clutter/ # GraspClutter6D
│ ├── dataset_omni6d/ # Omni6D
│ ├── dataset_omni3d/ # Omni3D (train/val/test)
│ ├── dataset_lerobot/ # LeRobot grouped configs
│ └── dataset_meta/ # Per-source sample lists (json)
│
├── scripts/
│ ├── launch/ # Training launch scripts
│ │ ├── pretrain.sh # Multi-GPU launch for pre-training (train_pretrain.py)
│ │ └── posttrain.sh # Multi-GPU launch for post-training (train_posttrain.py)
│ ├── download/ # Dataset download / extract / yaml-gen scripts
│ │ ├── agibot.sh
│ │ ├── interndata_a1.sh
│ │ ├── interndata_a1_unzip.sh
│ │ └── interndata_a1_generate_yaml.py
│ └── stats/ # Dataset normalization stats
│ ├── compute_dataset_stat_hdf5_abs_joint.py
│ ├── compute_dataset_stat_hdf5_rel_ee.py
│ ├── norm_robotwin.py # Robotwin EEP / qpos normalization stats
│ └── normalize.py
│
├── bin_stats/ # Non-uniform tokenizer bin boundaries
│ └── nonuniform_bins.pkl # Pre-computed quantization bins for 3D tokenization
│
├── docs/ # Stage-level documents
│ ├── PRETRAIN.md # Pre-train guide
│ ├── POSTTRAIN.md # Post-train / fine-tune guide (Robotwin)
│ └── ROBOTWIN_DATA.md # Robotwin raw → HDF5 conversion guide
│
├── robotwin/PoseVLA/ # RoboTwin simulation deploy + eval
└── google/paligemma-3b-pt-224/ # Local PaliGemma tokenizer / config
git clone git@github.com:hetolin/PoseVLA.git
cd PoseVLAInstallation details (click to expand)
- Python 3.10.12
- PyTorch 2.7.0 + CUDA 12.6 (TF32 / bf16 works best on Ampere / Hopper GPUs)
conda create -n vla python==3.10.12
conda activate vla
# Install PyTorch first — it is intentionally NOT pinned in requirements.txt,
# because the correct wheel depends on your local CUDA version.
# CUDA 12.6:
pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 \
--index-url https://download.pytorch.org/whl/cu126
# Install lerobot (pinned commit, compatible with PoseVLA)
# Fix: rename `pyav` -> `av` (package renamed on PyPI)
# Fix: remove `rerun-sdk` (requires numpy>=2, conflicts with imgaug)
git clone https://github.com/huggingface/lerobot.git third_party/lerobot \
&& cd third_party/lerobot \
&& git checkout 638d411cd3acf32c28d8c2120f3c41bda8bb15d4 \
&& sed -i 's/pyav/av/' pyproject.toml \
&& sed -i '/rerun-sdk/d' pyproject.toml \
&& pip install -e . \
&& cd ../..
# (Optional) Install bop_toolkit (used for BOP dataset, for 3D pretraining only)
git clone https://github.com/thodan/bop_toolkit.git third_party/bop_toolkit \
&& cd third_party/bop_toolkit \
&& pip install -e . \
&& cd ../..
# NOTE: keep the `lerobot` line commented out in requirements.txt — we install it manually.
pip install -r requirements.txt
Weights & environment setup (click to expand)
Create a pretrain/ directory under the project root and place any of the following weights (pick what you need — multiple loading branches are available in train_pretrain.py):
pretrain/
├── paligemma-3b-pt-224/ # Vanilla PaliGemma VLM
├── lerobot_pi0/ # π0 Action Expert
└── pi05_base/ # π0.5 Action Expert
Download links:
- PaliGemma-3B-pt-224 — Base VLM weights
- lerobot_pi0 — π0 Action Expert weights
- pi05_base — π0.5 Action Expert weights
The tokenizer is already bundled under google/paligemma-3b-pt-224/; the config points to it via model.tokenizer_model_path.
Set the project parent directory before launching Hydra-based training scripts. At minimum:
export DEV_PATH="/path/to/parent/of/PoseVLA"
export HYDRA_FULL_ERROR=1W&B auto-login: the script writes /root/.netrc — replace API_KEY with your own.
- For pre-training the PoseVLA backbone on large-scale 3D + action data, follow docs/PRETRAIN.md.
- For post-training / fine-tuning on RoboTwin (or your own robot), follow docs/POSTTRAIN.md.
- For the isolated PI0.5 RoboTwin baseline, follow pi05_baseline/README.md.
- For simulation evaluation on RoboTwin, follow robotwin/PoseVLA/README.md.
PoseVLA supports open-vocabulary 3D object grounding from RGB-D images. The inference script infer_grounding3d.py provides two modes:
python infer_grounding3d.pyWe provide example RGB-D images in assets/ for a quick demo:
| Input RGB | Input Depth | 3D Grounding Result |
|---|---|---|
 |
 |
 |
Modify the following in the script:
ckpt_path: path to your pretrained checkpointimg_path/depth_path: your RGB and depth image paths (default:assets/robot_rgb.png,assets/robot_depth.png)cam_K: camera intrinsic matrixtask_text: open-vocabulary query (e.g.,"detect the bottle")
The script will:
- Print the predicted 3D scene text and decoded structured result
- Save
infer_rgb.pngandinfer_depth_raw.pngfor inspection - Visualize 2D/3D detection results and depth/ray maps
- Release the training / co-training code for PoseVLA.
- Release the 3D evaluation entry (
eval_detection.py) with mAP / PR-curve aggregation. - Release support for both π0 and π0.5 Action Experts in a single codebase (switchable via
training.pi05). - Release the Robotwin post-training entry (
train_posttrain.py) and RoboTwin deployment scripts. - Release PoseVLA pretrain/posttrain checkpoints: Stage-1 (HuggingFace) | Robotwin (ModelScope).
- Release 3D grounding inference script (
infer_grounding3d.py) for real-world RGB-D input. - Release the runnable PI0.5 RoboTwin baseline with training, evaluation, and a ModelScope checkpoint.
🙋 FAQs
If you encounter any issues, feel free to open an issue on GitHub or reach out through discussions. Feedback and contributions are very welcome! 🚀
PoseVLA is built with reference to the following projects: lerobot, Transformers, Google PaliGemma, π0 / OpenPI, Omni3D, BOP Toolkit, and GraspClutter6D. Thanks for their awesome work.
@article{lin2026posevla,
title={PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies},
author={Lin, Haitao and Yu, Hanyang and Huang, Jingshun and Zhang, He and Ling, Yonggen and Tan, Ping and Xue, Xiangyang and Fu, Yanwei},
journal={arXiv preprint arXiv:2602.19710},
year={2026}
}