GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

ECCV 2026

Frédéric Fortier-Chouinard   Yannick Hold-Geoffroy   Valentin Deschaintre   Matheus Gadelha   Jean-François Lalonde

GimbalDiffusion provides gravity-aware camera control for video generation, including extreme pitch and roll trajectories expressed in an absolute world coordinate system. The method is implemented across multiple backbones—CogVideoX-2B, WAN 2.1, and WAN 2.2—and supports both Plücker-ray and UCPE camera encodings.

Set up the default WAN 2.1 1.3B + UCPE backbone, download its checkpoints, and generate a small hello-world set with every supported camera trajectory:

git submodule update --init --recursive
python3 -m venv .venv-ucpe
source .venv-ucpe/bin/activate
python -m pip install -r requirements-ucpe.txt

hf download Wan-AI/Wan2.1-T2V-1.3B \
  --local-dir checkpoint/backbones/Wan-AI/Wan2.1-T2V-1.3B
hf download lefreud/GimbalDiffusion gimbal_ucpe/dit.safetensors \
  --local-dir checkpoint/gimbaldiffusion

prompt="A dramatic canyon landscape with towering red rock cliffs carved by a winding river far below, warm sunset light illuminating layered rock formations, sparse desert vegetation clinging to the edges of the cliffs, vast open sky with glowing clouds above the horizon, rugged stone textures and dry shrubs in the foreground, ultra-detailed, cinematic lighting, natural colors, photorealistic, 35mm photography, depth of field, high dynamic range"
python -m src.scripts.inference_gimbal \
  --model gimbal_ucpe \
  --preset all \
  --prompt "$prompt" \
  --seed 246863 \
  --output-dir outputs/quickstart

The supported trajectories are backflip, pitch_up, pitch_down, roll, yaw_left, yaw_right, translate_right, translate_left, translate_down, translate_up, translate_forward, translate_backward, roll_forward, zoom_in, and zoom_out. The videos and matching 3D camera-grid visualizations are written to outputs/quickstart/. See the table below for more backbones, complete setup and training instructions, and the full inference options.

Experiment settings live in configs/, with one YAML file per released checkpoint.

Training-data download instructions are included in every training guide above. Evaluation datasets are downloaded as part of the workflow below.

GimbalDiffusion includes two evaluation entry points: SpatialVID-Extreme tests diverse absolute orientations and camera motion, while Prompt Entanglement tests whether a model follows pitch conditioning when it conflicts with the scene prompt. First complete the matching inference guide so the environment, backbone, and GimbalDiffusion checkpoint are available:

The commands below evaluate the default UCPE model. Activate a different environment and replace ours_ucpe with the corresponding method argument to evaluate another backbone.

Results correction (July 23, 2026). The original paper evaluated the Ours UCPE and Ours AC3D rows with each backbone's old negative prompt, whereas the WAN 2.1 and WAN 2.2 rows already used our intended look-down negative prompt. This affected inference only, not training. GimbalDiffusion inference now defaults to ours_look_down; pass --negative-prompt old only to reproduce the original no-look-down runs. Outputs generated this way use the explicit _old_negative_prompt suffix.

1. Download both benchmarks:

source .venv-ucpe/bin/activate

hf download lefreud/GimbalDiffusion-SpatialVID-Extreme spatialvid_extreme.tar \
  --repo-type dataset --local-dir data/spatialvid_extreme
tar -xf data/spatialvid_extreme/spatialvid_extreme.tar \
  -C data/spatialvid_extreme

hf download lefreud/GimbalDiffusion-Prompt-Entanglement prompt_entanglement.tar \
  --repo-type dataset --local-dir data/prompt_entanglement
tar -xf data/prompt_entanglement/prompt_entanglement.tar \
  -C data/prompt_entanglement

2. Generate the benchmark outputs:

python -m src.scripts.evaluate_spatialvid_extreme \
  --method ours_ucpe
python -m src.scripts.evaluate_text_entanglement \
  --method ours_ucpe

Generated videos are written as <sample-id>_out.mp4 under outputs/eval_spatialvid_extreme/assets/ours_ucpe/ and outputs/eval_text_entanglement/assets/ours_ucpe/. Each run uses the released prompt, seed, intrinsics, and the absolute or relative pose convention required by the selected model.

3. Compute the five per-video metrics reported in the paper table:

deactivate
python -m venv .venv-evaluation
source .venv-evaluation/bin/activate
python -m pip install -r requirements-evaluation.txt
mkdir -p checkpoint/evaluation
wget -O checkpoint/evaluation/i3d_torchscript.pt \
  'https://www.dropbox.com/s/ge9e5ujwgetktms/i3d_torchscript.pt?dl=1'

python -m src.scripts.evaluate_spatialvid_metrics \
  --data-root data/spatialvid_extreme \
  --videos-dir outputs/eval_spatialvid_extreme/assets/ours_ucpe \
  --output-csv outputs/eval_spatialvid_extreme/ours_ucpe_metrics.csv \
  --distribution-csv outputs/eval_spatialvid_extreme/ours_ucpe_fid_fvd.csv

This computes PitchErr and GravityErr with PerspectiveFields, relative rotation and translation errors with VGGT-1B, CLIPScore with CLIP ViT-L/14, Inception-2048 FID, and StyleGAN-V I3D FVD. For a quick validation, omit --distribution-csv and add --limit 2; FID/FVD are meaningful only on all 138 samples. To compute only FID/FVD, omit --output-csv. Add --reference-csv PATH and --reference-distribution-csv PATH to print matched means and deltas against existing result files. See the paper evaluation protocol and supplementary evaluation details.

4. Score Prompt Entanglement and plot the downloaded ground truth against your generated outputs:

python -m src.scripts.evaluate_text_entanglement_metrics score \
  --data-root data/prompt_entanglement \
  --method ground_truth_pitched=data/prompt_entanglement/assets/ground_truth_pitched \
  --method ours_ucpe=outputs/eval_text_entanglement/assets/ours_ucpe \
  --output-csv outputs/eval_text_entanglement/metrics.csv \
  --device cuda \
  --batch-size 16

python -m src.scripts.evaluate_text_entanglement_metrics plot \
  --metrics-csv outputs/eval_text_entanglement/metrics.csv \
  --paper-dir outputs/eval_text_entanglement/plots/paper \
  --supp-dir outputs/eval_text_entanglement/plots/supp

The scorer evaluates exactly 380 samples for each selected method. It batches and normalizes the semantic and pitch-adjusted CLIP text features once, then scores each first frame against both aligned feature sets.

@inproceedings{fortierchouinard2026gimbaldiffusion,
  title = {GimbalDiffusion: Gravity-Aware Camera Control for Video Generation},
  author = {Fortier-Chouinard, Fr{\'e}d{\'e}ric and Hold-Geoffroy, Yannick and Deschaintre, Valentin and Gadelha, Matheus and Lalonde, Jean-Fran{\c{c}}ois},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year = {2026}
}

Original code in this repository is MIT licensed. Submodules and model/data artifacts keep their own terms; see LICENSES.md.