Jan Ackermann Shengqu Cai Boyang Deng Zhengfei Kuang Songyou Peng Gordon Wetzstein

TL;DR: GeoFlow introduces a geometry-consistency reward — separating rigid camera-induced flow from dynamic object motion via optical flow and depth-pose predictions — and uses it to reinforcement-finetune video diffusion models towards physically consistent motion.

This repository contains a standalone sampler for generating videos with the GeoFlow LoRA on top of Wan2.1.

geoflow/                      # the geoflow package (pip install -e .)
├── rewards.py                #   metric factories (epipolar, met3r, geo, ...)
├── reward_server.py          #   the GeoFlow reward — multi-GPU DA3+WAFT server
├── reward_client.py          #   smoke-test client for the server
└── ...                       #   individual scorer implementations
scripts/
├── inference/sample_videos.py   # sample videos with the GeoFlow LoRA
└── eval/evaluate_videos.py      # score videos with the paper metrics
tests/                        # unit tests for the GeoFlow reward (CPU-only)
third_party/                  # pinned submodules: met3r, Depth-Anything-3, WAFT

• Install
• Model Weights
• Sampling
• CLI Reference
• Output
• Evaluation
• GeoFlow Reward
• Release Checklist
• Citation

We recommend a fresh conda/venv environment:

# Clone with submodules (external research code is vendored under third_party/)
git clone --recurse-submodules https://github.com/geometryflow/GeoFlow.git
cd GeoFlow

# Create and activate environment
conda create -n geoflow python=3.10
conda activate geoflow

# Install the geoflow package — covers sampling and the
# epipolar / raft_flow / gemini_* / geo_remote metrics
pip install -e .

# Extras for running the reward server (see "GeoFlow Reward"):
pip install -e '.[server]'

All external research code is vendored as pinned git submodules under third_party/ (nested submodules — MASt3R/DUSt3R/CroCo, DINOv3 — resolve automatically with --recursive):

They are only required for the components you use — sampling and the epipolar/raft_flow/gemini_* metrics work without any of them:

git submodule update --init --recursive   # if not cloned with --recurse-submodules

# `met3r` metric — MEt3R (MASt3R backbone)
pip install third_party/met3r

# `geo` metric (ours) — Depth Anything 3
pip install third_party/Depth-Anything-3

Note: MEt3R builds PyTorch3D and FeatUp from source, which requires a CUDA toolkit (nvcc) matching your PyTorch wheel and takes a few minutes.

The GeoFlow LoRA checkpoint is hosted on Hugging Face at ackermannj/geoflow:

huggingface-cli download ackermannj/geoflow --local-dir checkpoints/geoflow

The Wan2.1 base model (wan-ai/Wan2.1-T2V-1.3B-Diffusers) is downloaded automatically on first use.

Create a prompt file with one prompt per line:

printf "A camera moves forward through a narrow stone alley at sunset.\n" > prompts.txt

The sampler supports .txt prompt files (one prompt per line) and .jsonl files containing a "prompt" field per line.

Run the sampler with accelerate:

accelerate launch --num_processes 1 scripts/inference/sample_videos.py \
    --model_path wan-ai/Wan2.1-T2V-1.3B-Diffusers \
    --lora_path checkpoints/geoflow \
    --prompt_file prompts.txt \
    --output_dir generations/geoflow \
    --height 480 \
    --width 832 \
    --num_frames 81 \
    --num_inference_steps 50 \
    --mixed_precision bf16

Note — multi-GPU sampling: Increase --num_processes to sample on multiple GPUs; prompts are split across processes automatically. Omit --lora_path to sample from the unmodified Wan2.1 base model (e.g. for baseline comparisons).

Videos are saved to --output_dir as <prompt_index>_noise<noise_index>.mp4. After sampling completes, a prompts.json file mapping each video filename to its prompt is written to the same directory.

scripts/eval/evaluate_videos.py scores a directory of videos with the metrics used in the paper:

• epipolar — matches SIFT keypoints between consecutive frames, fits a fundamental matrix, and reports the negative mean Sampson distance. Higher is better.
• met3r — negative MEt3R multi-view consistency score (MASt3R backbone, DINO features) over consecutive frame pairs. Higher is better. Requires the third_party/met3r install (see Install).
• geo (ours) — regime-aware geometric-consistency reward built on Depth Anything 3: the prompt is classified into a motion regime (camera orbit/pan/forward, object turn, articulated, multi-object, static), and the video is scored with the matching reprojection-error or depth-consistency metric. Requires the third_party/Depth-Anything-3 install and prompts (--prompt_file).
• raft_flow — mean optical-flow magnitude between adjacent frames from torchvision's RAFT-Large; measures the dynamic degree of a video.
• gemini_consistency, gemini_semantic, gemini_dynamic — Gemini 2.0 Flash judge scores for temporal/visual consistency, semantic consistency, and dynamic degree, each normalised to [0, 1]. Require a GOOGLE_API_KEY env var; run on CPU.

Pass video files (or directories, or shell globs) directly:

python scripts/eval/evaluate_videos.py my_video.mp4 --metrics epipolar

python scripts/eval/evaluate_videos.py clips/*.mp4 \
    --metrics epipolar,met3r \
    --output_file evaluation_results.jsonl

accelerate launch --num_processes 1 scripts/eval/evaluate_videos.py \
    --video_dir generations/geoflow \
    --metrics epipolar,met3r \
    --output_file evaluation_results.jsonl \
    --mixed_precision bf16

# Our reward (needs prompts for regime classification)
accelerate launch --num_processes 1 scripts/eval/evaluate_videos.py \
    --video_dir generations/geoflow \
    --metrics geo \
    --prompt_file generations/geoflow/prompts.json \
    --output_file evaluation_geo.jsonl

As with sampling, increase --num_processes to evaluate on multiple GPUs; videos are split across processes automatically.

• Videos — any .mp4, .avi, .mov, .mkv, or .gif that ffmpeg can decode (RGB; any resolution and frame count). Directories are scanned non-recursively. Videos generated by scripts/inference/sample_videos.py work out of the box.
• Prompts — only needed for the geo and gemini_* metrics (epipolar, met3r, and raft_flow ignore them). Resolved per video, in order of priority:
  1. --prompt_file prompts.json — a JSON object mapping video filename (basename, not path) to prompt; the sampler writes exactly this format: {"0_noise0.mp4": "A camera moves forward...", ...}
  2. A companion text file next to the video (my_video.mp4 → my_video.txt) containing the prompt
  3. --prompt "..." — a single fallback prompt applied to all remaining videos
• Categories (optional) — --categories_file maps category names to prompt index ranges [start, end), e.g. {"static": [0, 50], "dynamic": [50, 100]}. This requires the sampler's filename convention (<prompt_index>_noise<k>.mp4), since the prompt index is parsed from the numeric filename prefix.

Per-video scores are written to --output_file (JSONL, one object per video), and aggregated global (and optionally per-category) averages are printed and saved to <output_file>_summary.json.

geoflow/reward_server.py is the reward model GeoFlow was trained with: a multi-GPU HTTP server that scores videos for geometric consistency. Per consecutive frame pair it computes

R = ½·R_geo + ½·R_dino

where R_geo compares the WAFT optical flow against the rigid flow induced by Depth Anything 3 depth + camera poses (plus a depth-warp consistency term), and R_dino is the negative cosine distance between DINOv2 features of the flow-warped source frame and the target frame. Both backbones run once per GPU worker and all scoring is batched on GPU.

pip install -e '.[server]'
pip install third_party/Depth-Anything-3

# WAFT checkpoints (not in the submodule):
# 1. The a1 `tar-c-t.pth` flow checkpoint, from the WAFT release
#    (https://github.com/princeton-vl/WAFT — "a1 adaptation" Google Drive link)
mkdir -p checkpoints/waft   # → checkpoints/waft/tar-c-t.pth
# 2. The Depth-Anything-V2-Small backbone WAFT was built on
mkdir -p third_party/WAFT/depth-anything-ckpts
wget -P third_party/WAFT/depth-anything-ckpts \
    https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth

DA3 (depth-anything/DA3-LARGE-1.1) and DINOv2 (facebook/dinov2-base) download automatically from Hugging Face on first start.

# One worker per GPU
python -m geoflow.reward_server --num-workers 8 --port 18090

# Smoke-test it from any machine
python -m geoflow.reward_client --video my_video.mp4 --fps 4.0 \
    --server http://localhost:18090

The protocol is HTTP + pickle: POST /score with {"videos": [[png_bytes, ...], ...], "fps": ...} returns per-video mean_R plus per-pair r_geo/r_dino details; GET /health reports worker status.

The reward math and geometry pipeline are covered by unit tests (no GPU or checkpoints needed): pip install -e '.[test]' && pytest.

The geo_remote metric in the evaluation script sends videos to a running server (no local GPU needed):

GEO_REWARD_SERVER_URL=http://<server>:18090 \
python scripts/eval/evaluate_videos.py generations/geoflow \
    --metrics geo_remote --output_file evaluation_geoflow_reward.jsonl

GEO_REWARD_FPS (default 8) and GEO_REWARD_SOURCE_FPS (default 16) control client-side frame subsampling before upload.

• Release model weights
• Release inference code
• Release eval code
• Release reward model code

@article{ackermann2026geoflow,
    author={Ackermann, Jan and Cai, Shengqu and Deng, Boyang and Kuang, Zhengfei and Peng, Songyou and Wetzstein, Gordon},
    title={GeoFlow: Enforcing Implicit Geometric Consistency in Video Generation},
    journal={arXiv preprint arXiv:2605.18365},
    year={2026}
}

• METHOD.md — current state of the reward method (start here).
• docs/reward_v3/ — the full v3 campaign: plans, Codex reviews, experiment protocols, and results (negatives included).
• docs/HANDOFF.md — v2-era project handoff and validation loop.