VisNavKit

A composable toolkit for visual navigation policies: train, export to ONNX, benchmark.

Vision in, trajectory out. Every stage is a Hydra group exchanging tokens of one width, so any encoder works with any goal, extra input and decoder. Ego state and camera calibration ship as inputs; anything else (depth, LiDAR, a spatial raster) is one encoder subclass plus a yaml.

context   vision   (B, F, 3, H, W)                            ─┐
          ego      (B, F, E)                                  ─┤
          camera   K (B, F, 3, 3), RT (B, F, 4, 4)            ─┼──▶  policy  ──▶  actions (B, M, T, D)
          <yours>  (B, F, ...)                                ─┤
goal      point · gps · image · route · instruction · <yours> ─┘

B batch, F frames, M candidates, T steps, D pose; every input but vision is optional. That function is policy.act(vision, goal, **inputs); forward is its training view (one decision per frame, packed for the losses) and predict its streaming view (the newest frame plus a feature buffer).

Architecture · Data · Models & weights · Benchmark · Roadmap

uv sync                    # CPU ONNX Runtime included; Linux resolves CUDA 13 torch wheels
uv sync --extra export     # + onnxslim for deployment graphs
uv sync --extra dali       # + NVIDIA DALI GPU video decoding (Linux)
export VISNAVKIT_DATA_ROOT=/data/nav_clips   # what common.data_root reads

Python 3.10+; the TorchCodec loader needs FFmpeg on the system (apt install ffmpeg).

No data, no downloads. Composes the pipeline, prints its shapes, runs a training forward and backward, and checks that act, forward and the feature-buffer predict agree on the newest decision; --onnx also exports and verifies ONNX Runtime parity:

uv run visnavkit-sanity-check model=s2e model/action_decoder=anchor_flow_dit --onnx

One entry per group, or a recipe plus overrides:

uv run visnavkit-train dataset=torch model=vint \
  model/vision_encoder=dinov3_s model/goal_encoder=gps model/action_decoder=flow_dit

Token modes, action spaces, per-signal normalizers and the denoiser × scheduler split behind the generative decoders: architecture.

Paper-named recipes adapt each architecture to this repo's data contract (single RGB frames, fixed-horizon x/y/v targets, goals from the episode's own future). They are not reproductions and load no upstream checkpoint; model=base is the skeleton they inherit.

One row per deployable variant: the paper authors' official weights, a VisNavKit reproduced-ckpt, and its visnavkit-export reproduced-onnx. Nothing is published yet.

uv run visnavkit-train dataset=torch model=gnm model/goal_encoder=point \
  ~model.goal_encoder.backbone_name ~model.goal_encoder.stack_observation  # gnm-point
uv run visnavkit-train dataset=torch model=flowpilot                       # flowpilot-edge
uv run visnavkit-train experiment=flowpilot_dst_clips1k                   # flowpilot-dst-small

Where each paper publishes its own weights, and the published ONNX graphs the benchmark downloads: model catalog.

A clip is a directory with video.mp4 and four NumPy sidecars (times, positions, orientations, speeds); a manifest lists video_path label start end. Targets are interpolated at fixed relative times, so any frame rate works.

uv run visnavkit-dataset command=preprocess                         # validate clips, write manifests
uv run visnavkit-dataset command=stats   dataset=torch name=city    # normalizer NPZ + plots
uv run visnavkit-dataset command=anchors dataset=torch num_anchors=64

Sidecar layout, the opt-in ego and calibration inputs, and the public corpora this format targets: data guide.

uv run visnavkit-train dataset=torch model=mimic ema=default
uv run visnavkit-train-route route=vae                      # route-patch AE/VAE from route_images.npy
uv run visnavkit-export checkpoint=logs/baseline/.../last.ckpt output=outputs/policy.onnx
uv run visnavkit-benchmark command=export model=gnm output_dir=outputs/benchmark/gnm

A route checkpoint seeds the route goal encoder: model/goal_encoder=route_image model.goal_encoder.weights=<ckpt>. Training records Git provenance (strict_git=true requires a clean tree) and keeps overrides in configs/experiment/. The deployment graph streams one frame through a feature buffer; the benchmark graph runs the full window for latency and open-loop metrics — architecture, benchmark.

uv run pytest tests/ -q && uv run ruff check visnavkit tests

Use it from another project with uv add --editable /path/to/visnavkit; the configs ship in the package as visnavkit.configs. Coding-agent contract: AGENTS.md.

VisNavKit is built on the following amazing open-source projects:

• PyTorch Lightning Training loop, checkpointing and callbacks.
• Hydra + OmegaConf Composable configuration for every stage.
• timm Every vision backbone, pretrained and feature-ready.
• TorchCodec and NVIDIA DALI CPU and GPU video decoding.
• ONNX Runtime and OnnxSlim Deployment graphs, parity checks and benchmarks.
• uv + Ruff Environments, linting and formatting.

The following repositories greatly inspire VisNavKit:

• diffusers — typed outputs, denoiser and scheduler as separate objects, the EMA schedule.
• openpi — flow-matching conventions and its Beta time sampling.
• LeRobot — one training forward, one deployment predict.
• diffusion_policy — the conditional 1D U-Net denoiser.
• visualnav-transformer — the GNM / ViNT / NoMaD recipes.
• openpilot — quadratically spaced trajectory anchors.

Thanks to the maintainers of these projects for their contribution to the community!

If VisNavKit helps your work, please consider citing it:

@Misc{visnavkit2026,
  author       = {Honglin He and Bolei Zhou},
  title        = {{VisNavKit}: a composable toolkit for visual navigation policies},
  howpublished = {\url{https://github.com/VAIL-UCLA/visnavkit}},
  year         = {2026},
}

CITATION.cff carries the same entry for GitHub's Cite this repository button; CITATION.bib has one entry per paper above — please also cite the work a recipe adapts.