Proxy Policy Steering
Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task-specific behavior from limited demonstrations without degrading their broad capabilities. We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge by training two lightweight proxy policies whose calibrated velocity-space difference steers the frozen base sampler. A reference proxy models the frozen base's behavior on target-task observations, and a task proxy, initialized from the reference, captures how this behavior changes under task supervision. Their difference forms a calibrated velocity-space residual that steers the frozen base sampler at every denoising step. We identify the conditions under which this residual isolates the change induced by task supervision, and validate them empirically. Because the base is never directly modified, its broad capabilities remain available at inference, including behaviors such as recovery from failure that the demonstrations themselves do not exercise. Adaptation requires only forward velocity predictions from the base, making PPS lightweight to train and applicable even without access to the base's parameters. On 8 real-world and 4 simulation manipulation tasks, PPS lifts the state-of-the-art π0.5 base policy by 55% absolute success rate on average, with zero-to-one gains on tasks the base never solves, while preserving the base's broad capabilities. PPS outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.
Our key insight is that effective adaptation should neither steer only within the base's existing distribution nor replace the base with a narrowly trained specialist. Instead, PPS estimates the direction in which task supervision would shift the base's action distribution, leaving its weights and broad priors untouched.
To estimate that direction without access to base gradients, PPS trains two lightweight proxy policies and takes their difference in velocity space.
- Reference proxy πref: distilled from πbase on the observations from task-specific demonstrations, mimicking the base behavior on those observations.
- Task proxy πtask: initialized from πref and fine-tuned on the task demonstrations. It differs from the reference policy through task supervision alone, and their difference in velocity space isolates the task-induced change.
At inference, PPS integrates the velocity predictions vi of the three policies πi in each flow-matching step, with coefficient γ representing the steering strength:
Here we show three representative cases: a state where the base acts differently from the demonstrations (residual steers toward the task behavior), a state not covered by the demonstrations (residual cancels, base prior recovers), and a state where the base already agrees with the demonstrations (residual vanishes).
We evaluate PPS on 8 real-world and 4 simulation manipulation tasks, spanning long-horizon goals, articulated and deformable objects, and diverse skills. PPS delivers consistent gains over the base model and outperforms fine-tuned models, from-scratch specialists, and prior steering baselines, with zero-to-one gains on tasks the base never solves on its own.
Select a task below to compare PPS rollouts with the frozen base policy and the strongest baselines on the same task.
Because PPS leaves the base model untouched, the combined policy inherits its broad priors and recovers from failures it never saw in the demonstrations. LoRA fine-tuning, by contrast, overfits to the handful of demonstrations and overwrites parts of these priors, breaking down under the same perturbations.
Drag the steering strength γ and watch the trade-off between base priors and task guidance.
Because the proxies are separate networks, they can take inputs the base never receives. PPS routes extra modalities such as point cloud or audio into the frozen, RGB-only base through the task proxy: the point cloud locates a textureless object, and audio tells the ringing phone from the silent one. This makes PPS a lightweight way to add new sensors without retraining the base.
Through a series of analyses, we study what makes PPS effective and where it still falls short.
Steering strength. The coefficient γ trades base priors against task steering. Success peaks for γ in (0.4, 0.6), well above both baselines.
Data scaling. On the two lowest-success tasks (Tea and Capsule), PPS leads at every demonstration budget from 50 to 200, with its largest margin in the low-data regime. The from-scratch specialist improves steadily and catches up near 200 demonstrations.
Failure modes. PPS fails far less often than the specialist and base, cutting incorrect-mode errors 93% versus the base and out-of-distribution errors 87% versus the specialist.
Proxy training. On-policy distillation, velocity-level supervision, and reference initialization each ensure the two proxies differ by task supervision alone, and their difference is an informative task signal.
@inproceedings{ning2026proxy,
title = {Proxy Policy Steering},
author = {Ning, Chuanruo and Wang, Tianrui and Ma, Wei-Chiu and Fang, Kuan},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}