LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

Vision-language models (VLMs) struggle with perspective taking: when a question asks about spatial relations from another entity's or an imagined observer's viewpoint, they often fall back to the camera view. LeRF trains a VLM to build and use an explicit entity-centered reference frame. Given an image and a question, the model decides whether a frame is needed. If it is, the model grounds the reference entity and predicts the frame's projected origin and its front / left / up axes. A lightweight renderer draws the frame on the image, and the model reasons over this visual cue. No external perception models or 3D reconstruction are used.

• Inference. Turn 1 (thinking off) produces either a draw_reference_frame call (reference entity, origin and three axis endpoints in [0, 1000] coordinates) or NO_TOOL_CALL. Turn 2 (thinking on) sees the rendered frame instead of the numeric coordinates, reasons over it and answers in \boxed{}.
• Stage I: SFT. The model learns reference-entity grounding and projected frame prediction from pose datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM). About 20% of the data are no-tool-call examples that teach selective tool use.
• Stage II: RL (GRPO). Trained on MultihopSpatial and SpatialReasoner-RL with a binary final-answer reward. The frame-prediction turn is excluded from the policy gradient, so only the reasoning and answer turn receives the advantage.

Accuracy (%) on OmniSpatial perspective taking (OmniSpatial-PT), 3DSRBench (Orientation / Multi-Object) and ViewSpatial-Bench (person-perspective Object View Orientation / Relative Direction). Bold marks the best open-source result; proprietary models are listed for reference.

LeRF-9B also beats every proprietary model listed on OmniSpatial-PT Hypo and ViewSpatial P-Obj. See the paper for more baselines, reference-frame estimation accuracy and ablations.

LeRF uses Qwen3.5-4B / 9B backbones. SFT (LoRA) was done with LLaMA-Factory. This repository contains the tool, the inference client and the GRPO training code built on verl.

LeRF/
├── tool/                  # the tool and inference-time tool calling
│   ├── frame_tool.py      # renders the frame (pure PIL), argument validation
│   ├── prompts.py         # tool schema, prompts and shared constants
│   ├── system_prompt.txt
│   └── inference.py       # two-turn tool-calling client for a vLLM server
├── verl/                  # verl (commit bf48903d) + our changes, see below
│   ├── lerf/              # agent loop, tool wrapper, reward, data preparation, tests
│   └── examples/lerf/run_grpo.sh
├── assets/                # README figures
└── requirements.txt

The environment is the standard verl environment (FSDP + vLLM) with a few extras.

conda create -n lerf python=3.12 -y && conda activate lerf
pip install vllm==0.24.0            # brings a matching PyTorch; other verl-supported versions may work
pip install flash-attn --no-build-isolation   # or a prebuilt wheel matching your torch / CUDA
pip install -r requirements.txt     # verl requirements + LeRF extras
pip install --no-deps -e verl
pip install causal-conv1d --no-build-isolation   # optional, faster Qwen3.5 linear attention

Tested with Python 3.12, CUDA 13.0, PyTorch 2.11, vLLM 0.24.0, transformers 5.10.4 on NVIDIA RTX PRO 6000 GPUs.

Serve the model with vLLM (OpenAI-compatible API), then run the client:

vllm serve /path/to/LeRF-9B --served-model-name lerf --max-model-len 32768 \
    --limit-mm-per-prompt '{"image": 2}' --trust-remote-code

cd tool
python inference.py --model lerf --image example.jpg \
    --question "From the man's perspective, which object is on his left?" \
    --options cup bed laptop phone --save_renders renders/

# batch: one JSON per line with image / question / options (and optionally answer, id)
python inference.py --model lerf --input questions.jsonl --output results.jsonl

Every record stores the full text of each turn (tool call, reasoning and answer), the parsed tool arguments and the prediction. Defaults follow the training setup: temperature 0.6, images downscaled to at most 1M pixels, a 10,240-token thinking budget within a 12,288-token answer turn.

1. Prepare data. The input is JSONL with image, question, options, answer (index or letter) and optionally id and data_source (one validation curve per data_source):

   cd verl
   python -m lerf.prepare_data --train train.jsonl --test test.jsonl --out_dir data/lerf

2. Train, starting from the tool-calling SFT checkpoint:

   MODEL_PATH=/path/to/sft_ckpt DATA_DIR=data/lerf N_GPUS=4 bash examples/lerf/run_grpo.sh

   Extra hydra overrides can be appended to the command. The defaults are our training setting (Qwen3.5-9B, 4 x 96 GB GPUs, batch 64 x 8 rollouts, lr 1e-6, KL 0.01).

How a rollout works (lerf/agent/frame_tool_agent_loop.py):

• turn 1 runs with thinking off and produces one draw_reference_frame call or NO_TOOL_CALL;
• turn 2 runs with thinking on from a rebuilt context: the tool call keeps only reference_object and is followed by the rendered image (or, for NO_TOOL_CALL, by a continuation instruction), so the frame image is the only geometry the model sees;
• each trajectory is stored as two training rows that share one reward; with algorithm.advantage_final_row_only=True only the answer row receives the GRPO advantage, while the tool-call row stays in the KL term;
• the reward (lerf/reward/reward.py) is 1 if the response contains exactly one \boxed{} matching the correct option, else 0.

Built on verl (Apache-2.0; see verl/LICENSE).

@misc{xiao2026lerflearningreferencecoordinate,
      title={LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning}, 
      author={Bang Xiao and Wenqi Jia and Ozgur Kara and Tiancheng Shen and Yibo Yang and Bolin Lai and Junho Kim and James Matthew Rehg},
      year={2026},
      eprint={2609.36219},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.36219}, 
}