1
2
3Real-world reinforcement learning for robotic manipulation remains challenging, and this difficulty is amplified for flow matching policies: applying policy gradient methods to these policies is fundamentally limited by the need to backpropagate through time (BPTT) along the multi-step ODE that maps noise to actions, which is computationally prohibitive and numerically fragile. We propose FlowDPG, a DDPG-style method specifically designed for flow matching policies that distills the critic gradient into the velocity field at training time, bypassing BPTT entirely. Intuitively, FlowDPG combines two complementary vectors: the demonstration-driven velocity that keeps the action feasible, and the critic-driven correction that steers it toward higher value. Our contributions are threefold: (1) a BPTT-free in-place framework that leaves the deployed multi-step inference path unchanged, (2) a formal connection between the FlowDPG update direction and vanilla Deterministic Policy Gradient via three explicit approximations, and (3) real-world validation on two long-horizon, multi-stage, dual-arm tasks on two different robots — AirPods assembly on a Franka and scrambled-egg cooking on a YAM.
AirPods Assembly on Franka
One hour uncut · 25 cases back to back · no human intervention
Scrambled Eggs on YAM
Recovery from Failures
Method Overview
FlowDPG vs. BC and QAM
50 rollouts per cell (10 at each of 5 checkpoints). Rubric: 0–3 per stage (criteria). Eggs: offline only.
All baseline families (AirPods)
25 cases per method; baselines trained offline only.
Human Disturbances
Recovery rate under human disturbances (AirPods)
Stage-Aware Reward Model
Reward ablation
case
case
R pod
R pod
L pod
L pod
case
case
Share of AirPods episodes that get through each stage.
Consistency Loss and Adaptive Shift
Swipe sideways for all four panels.
All variants include the online phase.
Evaluation rubrics
0–3 per stage · AirPods: 8 stages, 24 points · Eggs: 14 stages, 42 points
Click to expandCollapse
A stage never reached scores 0. Criteria were fixed before scoring.
AirPods assembly (Franka) · 8 stages, 24 points
| Stage | 3 – Successful | 2 – Minor error | 1 – Major error | 0 – Failed |
|---|---|---|---|---|
| Grasp case | Secure grasp, correct orientation, no slip | Grasp with adjustment or delay | Case lifted but shifted or dropped once | Fails to grasp case |
| Open case | Lid fully opened in one clean motion | Opens with adjustment or hesitation | Opens only after repeated attempts, or case displaced | Lid not opened |
| Grasp right pod | Secure grasp, correct approach direction | Grasp with adjustment or delay | Wrong approach direction, pod dislodged then re-grasped | Fails to grasp pod |
| Insert right pod | Seated on the first attempt, fully in | Seated after one retry | Seated after two or more retries, or insertion dislodges the other pod | Pod not seated |
| Grasp left pod | Secure grasp, correct approach direction | Grasp with adjustment or delay | Wrong approach direction, pod dislodged then re-grasped | Fails to grasp pod |
| Insert left pod | Seated on the first attempt, fully in | Seated after one retry | Seated after two or more retries, or insertion dislodges the right pod | Pod not seated |
| Close case | Lid closed cleanly with both pods seated | Closes after minor adjustment | Closes with a pod unseated, or needs repositioning | Case not closed |
| Place case | Placed upright within the target region | Placed with minor misalignment | Dropped or placed outside the region | Case not placed |
Scrambled eggs (YAM) · 14 stages, 42 points
Pick up and crack occur twice, once per egg: 14 scored stages from 12 types.
| Stage | 3 – Successful | 2 – Minor error | 1 – Major error | 0 – Failed |
|---|---|---|---|---|
| Pick up egg | Secure grasp, no slip or damage | Grasp with adjustment or delay | Lifts egg but drops or damages it | Fails to grasp egg |
| Crack egg | Clean crack, proper shell separation | Minor shell fragments or awkward motion | Incorrect or messy crack | Fails to crack egg |
| Put salt | Correct amount applied accurately | Minor spillage or hesitation | Incorrect amount or placement | Salt not applied, or bottle dropped |
| Pick up fork | Secure and correct grasp | Grasp with adjustment | Unstable or incorrect grasp | Fails to pick up fork, or drops it |
| Whisk egg | Thorough whisking with correct motion | Partial or inefficient whisking | Minimal or incorrect motion | No whisking |
| Place fork back | Placed neatly in correct location | Minor misalignment | Dropped or poorly placed | Fork not placed back |
| Put eggs into pan | Clean transfer into pan | Minor spillage or hesitation | Significant spillage | Eggs not transferred |
| Pick up spatula | Secure and correct grasp | Grasp with adjustment | Unstable or incorrect grasp | Fails to pick up spatula |
| Stir eggs | Proper stirring across pan | Incomplete or uneven stirring | Minimal or incorrect stirring | No stirring |
| Transfer eggs | Clean and accurate transfer | Minor spillage or inefficiency | Partial or messy transfer | Eggs not transferred |
| Place spatula back | Placed correctly and safely | Minor misplacement | Dropped or unsafe placement | Spatula not placed back |
| Serve eggs | Served correctly and fully | Minor error or delay | Incorrect or incomplete serving | Eggs not served |
BibTeX
@inproceedings{shi2026flowdpg,
title = {FlowDPG: Deterministic Policy Gradient on Flow Matching Policies
for Real-World Manipulation},
author = {Shi, Kexin and Shi, Junyao and Hebbar, Poorvi and Zhao, Zhuolun and
Amarnath, Tarun and Su, Yifan and Bahl, Shikhar and Pathak, Deepak},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}
