Generative Cinematographer

Jiahan Zhang · Chaohao Yang · Namitha Guruprasad · Vivekjyoti Banerjee
Trong-Tung Nguyen · Alan Yuille · Anand Bhattad
Johns Hopkins University

From a single image to a composed shot. Author camera paths and object motion together in 3D, then use a video diffusion model to generate the shot.

Input image → authored 3D controls → generated video.
Watch the overview · Explore the interactive demos

This repository provides offline inference with preprocessed inputs. The included examples generate 21 frames at 832×480 using the provided checkpoint. The demos above show authored motions from the project website; the bundled examples use their provided trajectories.

Environment · Download weights · Run inference · Examples · Citation

The full method lifts an image into a 3D scene, represents camera and object motion with correspondence maps, and conditions video generation on those maps. This release runs the inference stage using prepared image, text, and geometry conditioning. See the project website for the authoring workflow.

Allow about 50 GB of disk space for the checkpoint and enough host RAM to hold the weights during loading and CPU/GPU transfers. requirements.lock pins the runtime dependencies, including CUDA libraries and the video writer.

Clone the code repository, create an environment, and install the package:

git clone https://github.com/vanillaer/GenCine.git
cd GenCine
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.lock
python -m pip install --no-deps .
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Reinstall with python -m pip install --no-deps . after changing package code.

The --weights argument points to the bundle directory, with this layout:

weights/gencine/
├── model.json
├── dit.safetensors
├── vace.safetensors
├── scale3d.safetensors
└── vae_decoder.safetensors

The checkpoint repository is Gencine_ckpts and is public. Install the Hugging Face CLI in your download environment (python -m pip install huggingface_hub), then download the checkpoint:

hf download zonszer/Gencine_ckpts --revision 0c90925af792126ee26564fcdf840c09659ec18f \
  --local-dir weights/gencine

Pass the downloaded bundle directory to --weights. model.json contains the model configuration and SHA-256 checksums for all four weight files. The loader verifies these checksums and strictly loads all four model components. No checkpoint download is performed during inference.

Run one example on GPU 0:

CUDA_VISIBLE_DEVICES=0 python -m gencine \
  --weights weights/gencine \
  --manifest examples/manifest.json \
  --example hike \
  --output outputs/hike_21

Run all provided examples sequentially, reusing the loaded models:

CUDA_VISIBLE_DEVICES=0 python -m gencine \
  --weights weights/gencine \
  --manifest examples/manifest.json \
  --all \
  --output outputs/examples_21

For available examples, see example contents for their files and format.

The provided examples are prepared natively at 21 frames, with six latent frames in the noise, identity, and geometry tensors. Frame count comes from each example's metadata and encoded inputs.

Use --seed N to regenerate noise, or omit it to reuse the saved noise. Prompt text in metadata describes the cached embedding; editing the text alone does not change model conditioning.

Each run saves:

• video.mp4: generated RGB video.
• run.json: effective settings, hashes, environment, timing, and peak GPU memory.
• imported_modules.json: imported package paths.

The provided examples use 3D annotations estimated with CoWTracker/ViPE.

The runtime contains model loading, denoising, and RGB decoding. New examples need precomputed inputs; preprocessing and training are outside this package. See LICENSE and THIRD_PARTY_NOTICES.md for licensing.

If you use this project, please cite:

@misc{zhang2026generativecinematographer,
  title         = {Generative Cinematographer: Composing Camera and Object Motion in 3D},
  author        = {Zhang, Jiahan and Yang, Chaohao and Guruprasad, Namitha and Banerjee, Vivekjyoti
                   and Nguyen, Trong-Tung and Yuille, Alan and Bhattad, Anand},
  year          = {2026},
  eprint        = {2610.02180},
  archivePrefix = {arXiv}
}