FlowPilot

From Imitation to Alignment: Human-Preference Flow Policies for Long-Horizon Sidewalk Navigation

Conference on Robot Learning (CoRL) 2026

_{GPS-guided long-horizon navigation · sidewalk lane keeping · obstacle avoidance · pedestrian awareness · night driving}

_{Closed-loop rollouts in SidewalkBench-GS · Gaussian-splat reconstructions of real sidewalks}

The model code lives in VisNavKit, included here as a submodule on its dev branch.

git clone --recurse-submodules https://github.com/VAIL-UCLA/FlowPilot.git
cd FlowPilot/visnavkit
uv sync --extra export

In an existing clone, run git submodule update --init instead.

Weights are hosted in the VAIL model zoo on Hugging Face.

* The model with the old recipe, for experiments. It is goal-free and differs from the DST models: a pinhole camera at 512 x 288 (fx = fy = 210.667, cx = 256, cy = 144), 21 frames at 5 Hz (4 s of history) in, 64 modes of 20 [x, y] waypoints at 5 Hz out (drop mode 0 of the raw 65, a register token). flowpilot does not wrap it; use the zoo's inference.py and model_info_goal_free.yaml.

Inputs, outputs and a minimal ONNX Runtime example for the flowpilot-dst models are in the ONNX guide.

flowpilot wraps the ONNX exports in the interface of the Navigation Model Zoo: frames and a goal in, a plan and a (v, w) command out.

pip install -e .                 # from the FlowPilot root
pip install onnxruntime-gpu      # optional, replaces onnxruntime for CUDA (1.23.2 runs on CUDA 12 drivers; 1.31 needs CUDA 13)

import numpy as np
from flowpilot import FlowPilotNavigator, GpsGoal, ImprovedFlowPilotNavigator, Route, RouteGoal, compass_to_yaw

nav = FlowPilotNavigator(variant="flowpilot-dst-small", device="cuda")   # downloads the graph

obs = np.random.rand(1, nav.context_size, 3, 216, 384).astype(np.float32)   # (B, T, 3, H, W) in [0, 1]
ego_vw = [1.2, 0.0]     # measured speed (m/s) and yaw rate (rad/s)

# Point goal: metres in the ego frame, x forward, y left
vw, plan = nav.inference_vw(obs, [20.0, 0.5], ego_vw)                    # pure pursuit, vw: (B, 2)
vw, plan = nav.inference_vw(obs, [20.0, 0.5], ego_vw, controller="pd")   # PD
traj, scores = nav.inference_trajectory(obs, [20.0, 0.5], ego_vw)        # (B, 6, 80, 5), (B, 6)

# GPS goal: a waypoint and the robot's fix, converted to a point goal
goal = GpsGoal(goal=(34.06908, -118.44505), robot=(34.06893, -118.44520), yaw=compass_to_yaw(45.0))
vw, plan = nav.inference_vw(obs, goal, ego_vw)

# Route goal: the route patch around the robot, plus the route point 20 m ahead as the point goal
route = Route([[0.0, 0.0], [25.0, 0.0], [25.0, 40.0]], crosswalks=None)   # world metres; Route.from_gps for lat / lon
vw, plan = nav.inference_vw(obs, RouteGoal(route, pose=(3.0, 0.2, 0.0)), ego_vw)

vw, plan = nav.step(obs[0, -1], [20.0, 0.5], ego_vw)   # streaming: one frame per call at 20 Hz
nav.reset()                                            # between episodes

# improved_flowpilot: streaming only, no ego status, no point goal (the route patch guides it)
nav = ImprovedFlowPilotNavigator(device="cuda")
vw, plan = nav.step(obs[0, -1], RouteGoal(route, pose=(3.0, 0.2, 0.0)))   # every 20 Hz frame, in order

flowpilot-dst models: pass ego status and a goal at least 20 m away. They take the robot's measured speed and yaw rate, [v, w] for each of the 20 frames, as a graph input (ego, shape (B, 20, 2)): pass odometry as ego_vw. The goal encoder clips distance at 20 m, so a goal 20 m or farther reads as "keep driving", while a nearer one reads as arriving and the model slows to a stop. RouteGoal already sets its goal 20 m along the route; for a point or GPS goal, use a waypoint at least 20 m ahead unless the robot should stop there. improved_flowpilot needs neither: it reads only the frames and the route patch.

• Observations are 20 frames at 20 Hz, oldest first, at any size. Shorter histories are padded with their oldest frame.
• Plans hold [x, y, yaw, v, w] every 0.05 s up to 4 s, with modes ranked best first.
• Goals are required by the flowpilot-dst exports, which have no goal-free input. World poses are (x, y, yaw) with x east, y north and yaw counter-clockwise from east.
• Ego status should be measured odometry. Without it the previous command stands in, and a window of zeros reads as a robot at rest. A route goal with a pose per frame derives it from the poses.
• Controllers also run on their own: make_controller("pure_pursuit").step(plan, ego_speed=1.2) takes any [x, y] or [x, y, yaw, v, w] path. Limits, lookahead and gains are config fields, for example max_v, max_steering_angle=None for differential drive, kp and kd.

Use the training camera. The flowpilot-dst models and improved_flowpilot were trained on clips1k from a single camera model; other optics, mounts or crops shift the plans. We highly recommend using this standard configuration.

clips1k robots carry a small per-robot principal-point offset (principal_point_delta in each clip's camera.json), which training removes by shifting the frames; a camera centred at (cx, cy) above needs no shift.

python examples/run_clips.py mini_data/*/ [--variant flowpilot-dst-dune | improved_flowpilot]   # every frame from 19 on -> <clip>/<variant>.npz
python examples/render_clips.py mini_data/*/                                                     # one panel per model -> <clip>/flowpilot_viz.mp4

A clip holds front_fe_384x216.mp4, the frame_*.npy arrays at 20 Hz and route_full.json. The .npz holds the commands, plans, goals, route patches and ground truth. The video draws each model's 4 s plan as a cyan ribbon on the ground, projected through the camera above, with the route patch the model saw as an inset.

_{render_clips.py on clip 0973fd34 of mini_data · left to right flowpilot-dst-small, flowpilot-dst-dune, improved_flowpilot · inset: the route patch the model sees}

_{Clip c7d48ccc at night · a crosswalk, then a turn onto the sidewalk}

• Goal distance. The point-goal encoder clips distance at 20 m, so a goal 20 m or farther reads as "keep driving"; a nearer goal tells the model it is arriving, and it slows down. The route goal therefore sits 20 m ahead along the route (goal_ahead_m); its straight-line distance dips a little on curves (17 m on these clips) and drops only near the route's end.
• Route. route_full.json is a polyline of [lon, lat] points (plain coordinates or a GeoJSON geometry). Route.from_gps takes it as [lat, lon] in a local east-north frame, and the robot's pose per frame is frame_gps ([lat, lon]) with frame_yaw_deg (counter-clockwise from east). From the poses the route renders the 40 x 40 m patch per frame and gives the point goal.
• Ego status is frame_speeds and the yaw rate frame_angular_velocities[:, 2].
• Ground truth for scoring is the future of frame_positions / frame_orientations (wxyz) in the current ego frame.
• The principal-point shift is skipped here, since these clips carry no camera.json.

Run these from the visnavkit directory.

uv run visnavkit-train dataset=torch model=flowpilot                  # FlowPilot: FastViT-MA36, 64 anchors
uv run visnavkit-train experiment=flowpilot_dst_clips1k               # flowpilot-dst-small
uv run visnavkit-train experiment=flowpilot_dune_dst_clips1k \
  model.frame_encoder.weights=<DUNE ViT-B/14 ckpt>                    # flowpilot-dst-dune
uv run visnavkit-export-dst checkpoint=<ckpt> output=flowpilot_dst.onnx

The human-preference alignment stage will be released here.