Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs (H2RB)

H2RB combines geometry-grounded visual representations with a shared action latent space and independent human/robot reconstruction decoders. This repository provides an action-only tokenizer, deterministic latent caching, and RLDX-1-IMG fine-tuning with independent switches for geometry alignment and latent actions.

This is a code release. In-house demonstrations, task descriptions, task lists, episode inventories, normalization statistics, calibration, and trained weights are excluded. Supply your own canonical data and locally computed statistics.

Use Python 3.10, an NVIDIA driver compatible with CUDA 12.6, a CUDA 12.x development toolkit (nvcc), and a C++ compiler. FFmpeg shared libraries must be available for TorchCodec video decoding. The toolkit/compiler are needed when FlashAttention builds from source.

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip "setuptools>=67" wheel packaging ninja psutil numpy==1.26.4
python -m pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu126
MAX_JOBS=4 python -m pip install --no-cache-dir -e . --no-build-isolation
python -m pip check

Install PyTorch before the project: FlashAttention imports torch while preparing its package metadata. Build tools must also be installed explicitly because --no-build-isolation uses the active environment. Disabling the pip cache for this step also avoids FlashAttention wheel-renaming failures when cache and temporary directories are on different filesystems. The CUDA wheel pair follows the official PyTorch 2.7.0 instructions.

This implementation builds on the RLDX-1 and NVIDIA Isaac GR00T N1.7 codebases. This release uses RLDX-1-IMG as the policy backbone. Original upstream license and copyright notices are retained; see provenance. The vendored runtime lives in rldx/, while h2rb/ contains the portable method components. Tokenizer and adapter modules do not import the VLM runtime.

Prepare per-episode float32 action arrays [frames,15] in a canonical shared coordinate frame: wrist position (3), rotation 6D (6), and hand coordinates (6). Use an explicit episode-level train/validation split in a manifest following examples/tokenizer-manifest.json. normalization.json contains user-computed min and max vectors: coordinate-wise minimum and maximum across the human/robot training sources. Do not use validation or held-out evaluation episodes to fit these bounds.

bash scripts/train_tokenizer.sh \
  --manifest /path/to/manifest.json --stats /path/to/normalization.json \
  --output /path/to/tokenizer_run

The default is a shared four-block, width-256 encoder, 64 latent channels, two complete four-block decoders, horizon 16, VAE KL weight 1e-6, and position-wise cross-domain regularization weight 0.03. Set --reg-weight 0 for the no-regularizer ablation. Tokenizer optimization uses AdamW at 5e-5 with a constant schedule, weight decay 1e-5, and global batch 512 on two GPUs. Checkpoints contain optimizer state; rerunning resumes the latest saved checkpoint. The serving bundle is exported under tokenizer/. The training contract records the distributed regularizer implementation; legacy runs without this field require a fresh output directory instead of silent resume.

bash scripts/cache_latents.sh \
  --manifest /path/to/source_manifest.json --stats /path/to/normalization.json \
  --tokenizer /path/to/tokenizer_run/tokenizer --output /path/to/latent_cache

Each manifest must use the same episode order/indices as its corresponding policy dataset. The cache stores float16 [frames,16,64] posterior means, with end-repeat padding. It binds targets to hashes of tokenizer weights, normalization and source actions, and writes completion metadata after every episode succeeds. Policy loading additionally verifies the canonical episode action content against the cache. Rebuild older caches that do not contain source-content hashes.

Convert your data to canonical LeRobot v2 episodes with a shared action/state schema. Configure paths and mixing weights in a local copy of examples/policy-recipe.json. Both sources share an embodiment tag/projector; geometry sidecars contain current-frame labels and coordinate validity. See the exact data contract.

bash scripts/train_policy.sh \
  --recipe /path/to/local_recipe.json --output /path/to/policy_run \
  --alignment on --latent-actions on

Defaults: two GPUs, global batch 64, one optimizer update per batch, 30k steps, AdamW at 1e-4 with cosine decay and 5% warmup, weight decay 1e-5, bf16, seed 42, a frozen VLM, a 256-wide SiLU residual adapter, and geometry loss weight 0.2. Checkpoints are saved every 1k steps, retaining the latest two. Resume includes optimizer/scheduler state. Use a fresh output directory for a different recipe.

Use a trained checkpoint-N or final directory, its saved processor/, and one of your canonical LeRobot dataset roots. This example predicts from the first valid frame's image, current state, and task text; it removes action labels before calling the policy. The dataset schema and embodiment tag must match training. For a different schema, use the modality keys stored in the saved processor.

from pathlib import Path

import numpy as np
import torch

from h2rb.policy.action_model import install
from h2rb.policy.runtime import LatentActionDecoder
from rldx.data.dataset.standard_single_step_dataset import StandardSingleStepDataset
from rldx.data.embodiment_tags import EmbodimentTag
from rldx.model.core.processing_rldx import RLDXProcessor
from rldx.model.core.rldx import RLDX

checkpoint = Path("/path/to/policy_run/final")
dataset_root = Path("/path/to/canonical_source_a")
tokenizer = Path("/path/to/tokenizer_run/tokenizer")
tokenizer_stats = Path("/path/to/normalization.json")
tag = EmbodimentTag("general_embodiment")

install()
model = RLDX.from_pretrained(checkpoint, dtype=torch.bfloat16).eval().to("cuda")
processor = RLDXProcessor.from_pretrained(checkpoint / "processor")
processor.eval()
dataset = StandardSingleStepDataset(
    dataset_path=dataset_root,
    embodiment_tag=tag,
    modality_configs=processor.modality_configs[tag.value],
    video_backend="torchcodec",
    allow_padding=False,
)
dataset.set_processor(processor)
batch = processor.collator([dataset[0]])
batch["inputs"].pop("action", None)
batch["inputs"].pop("action_mask", None)
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    prediction = model.get_action(**batch)["action_pred"]

if model.config.h2rb_actlat_mode:
    decode = LatentActionDecoder(tokenizer, tokenizer_stats,
                                decoder_id="shared_prq__human")
    actions = decode(prediction).float().cpu().numpy()
else:
    decoded = processor.decode_action(prediction.float().cpu().numpy(), tag)
    keys = processor.modality_configs[tag.value]["action"].modality_keys
    actions = np.concatenate([decoded[key] for key in keys], axis=-1)
actions = np.asarray(actions, dtype=np.float32)
assert actions.shape[0] == 1 and actions.shape[1] == model.config.action_horizon
assert actions.shape[2] == 15 and np.isfinite(actions).all()
np.save("predicted_actions.npy", actions[0], allow_pickle=False)
print(actions[0].shape)  # (16, 15) with the default horizon

The saved array contains canonical physical wrist/hand targets. In latent mode, use the frozen tokenizer and bounds from training; select shared_prq for the robot reconstruction branch or shared_prq__human for the human branch. Both branches share the encoder. In physical-action baseline mode, the saved processor reverses action normalization directly. Hardware IK, robot calibration, and control remain part of the deployment integration.

python -m unittest discover -s tests -v

Tests use synthetic arrays only and cover encoder/export equivalence, independent decoders, regularizer reductions, cache identity and padding, geometry masks, actual A/B forward switches, and train/inference adapter parity. A queued or completed training run does not establish a real-robot transfer success rate.

The release accepts JSON recipes and numeric, non-pickled data. Legacy Python-object YAML configuration and pickled initial-action sidecars are not supported.

If you use this work, please cite our paper:

@misc{park2026humantorobottransferbridgingrepresentation,
      title={Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs},
      author={Junhyeong Park and Wook Jung and Taeyoung Kim and Huiwon Jang and Jinwoo Shin},
      year={2026},
      eprint={2611.04391},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2611.04391},
}