SpaceTools-RL

This repository contains the RL training and evaluation component of SpaceTools, a tool-augmented spatial reasoning system for VLMs. It is a fork of verl v0.8.0-dev with extensions for multi-turn tool interaction during RL training.

This repo is intended to be used as a submodule of the SpaceTools parent repository. See the parent repo for full pipeline instructions.

Core modifications for multi-turn tool interaction during RL:

examples/toolshed/
├── run_rl.sh                       # RL training (V1 or V2)
├── run_rl_roborefer.sh             # Roborefer-only RL (standalone, resumable)
├── run_eval.sh                     # Evaluation on paper benchmarks
├── generate_toolshed_config.py     # Auto-generate tool config from live actors
├── toolshed_v1_config.yaml         # V1 tool schemas (11 tools, no robot)
├── toolshed_v2_config.yaml         # V2 tool schemas (17 tools, with robot)
└── README.md                       # Toolshed integration details

analysis/
└── analyze_grasp_result.py         # Grasp evaluation analysis

GRPO training with live Toolshed tool execution. Takes an SFT checkpoint (produced by SpaceTools-SFT) as input.

Required environment variables:

Optional environment variables:

Usage:

conda activate spacetools-rl

SFT_CHECKPOINT=/path/to/sft_checkpoint \
ROBOREFER_MODEL=/path/to/RoboRefer-8B-SFT \
DEPTH_CHECKPOINT=/path/to/depth_pro.pt \
    bash examples/toolshed/run_rl.sh

# V2 (with mock robot):
VERSION=v2 SFT_CHECKPOINT=... ROBOREFER_MODEL=... DEPTH_CHECKPOINT=... \
    bash examples/toolshed/run_rl.sh

SLURM (multi-node, wall-time resubmission):

The script requires 2 nodes and must be launched via SLURM. Training typically takes 8-12 hours (~86 steps at ~5-7 min/step), which may exceed wall-time limits. The script auto-resumes from the latest checkpoint in OUTPUT_DIR, so you can resubmit the same job after a wall-time kill:

# Create a wrapper script that sets env vars and sources run_rl.sh,
# then submit via sbatch:
sbatch --nodes=2 --gpus-per-node=8 --exclusive --time=4:00:00 my_wrapper.sh

# After wall-time kill, resubmit the same script to resume:
sbatch --nodes=2 --gpus-per-node=8 --exclusive --time=4:00:00 my_wrapper.sh

See examples/toolshed/README.md for a full wrapper example.

Standalone RL training with roborefer detect_one only. Trains from base Qwen model (no SFT checkpoint required). Automatically resumes from existing checkpoints.

ROBOREFER_MODEL=/path/to/RoboRefer-8B-SFT \
    bash examples/toolshed/run_rl_roborefer.sh

Evaluates a model checkpoint on the paper's 9 benchmarks with live tool execution.

Usage:

# All benchmarks
ROBOREFER_MODEL=/path/to/model DEPTH_CHECKPOINT=/path/to/depth_pro.pt \
    bash examples/toolshed/run_eval.sh /path/to/model_checkpoint

# Specific benchmarks
bash examples/toolshed/run_eval.sh /path/to/model robospatial bopgrasp blinkdepth

Available benchmarks:

Custom reward functions in verl/utils/reward_score/:

RL training requires 2 nodes (16 GPUs total). Tool actors are packed onto one node via a Ray placement group, and training uses the full second node.

Node 1 — Tool actors (8 GPUs):

Node 2 — Training (8 GPUs):

FSDP training + sglang rollout engine, using trainer.n_gpus_per_node=8.

The script automatically detects the SLURM node allocation and starts a multi-node Ray cluster.

Uses the spacetools-rl conda environment. See the parent repo setup guide.

Key dependencies: sglang==0.5.6, ray==2.47.1, torch==2.9.1, flash-attn, numpy<2.0.0.

Critical: Set LD_LIBRARY_PATH before running any script:

export LD_LIBRARY_PATH="${CONDA_PREFIX}/lib:${CONDA_PREFIX}/lib/python3.11/site-packages/nvidia/cuda_runtime/lib:${CONDA_PREFIX}/lib/python3.11/site-packages/nvidia/cudnn/lib:${LD_LIBRARY_PATH:-}"

This repository is built on verl (Volcano Engine Reinforcement Learning for LLMs) by the ByteDance Seed team. We thank the verl team for their flexible and efficient RL training framework.

@inproceedings{sheng2024hybridflow,
    title={HybridFlow: A Flexible and Efficient RLHF Framework},
    author={Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
    booktitle={The Twentieth European Conference on Computer Systems (EuroSys)},
    year={2025},
    url={https://arxiv.org/abs/2409.19256v2}
}

@misc{chen2025spacetoolstoolaugmentedspatialreasoning,
    title={SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL},
    author={Siyi Chen and Mikaela Angelina Uy and Chan Hee Song and Faisal Ladhak and Adithyavairavan Murali and Qing Qu and Stan Birchfield and Valts Blukis and Jonathan Tremblay},
    year={2025},
    eprint={2512.04069},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2512.04069}
}