🪄 WeatherCrafter

Semantic-Aware, Physics-Informed, Geometry-Grounded
Weather Video Synthesis

Chenghao Qian¹  ·  Nedko Savov²  ·  Lingdong Kong³  ·  Yeying Jin³  ·  Rui Song⁵
Wenjing Li^1,4  ·  Zhun Zhong⁴  ·  Jiaqi Ma⁵  ·  Gustav Markkula¹  ·  Luc Van Gool²

We steer an off-the-shelf video diffusion editor with three structured priors — semantics (what the weather looks like), dynamics (how it evolves), and geometry (where it appears) to synthesize diverse, physically realistic weather on real videos, without any finetuning.

• 🧩 Tri-prior interface — a single, structured conditioning space that factorizes weather into semantics · dynamics · geometry, giving precise and interpretable control.
• 🌦️ Diverse appearance — a semantic-aware strategy binds the intended weather to scene semantics via a VLM + LLM, producing varied, realistic global appearances.
• ❄️ Physical particle dynamics — a physics-informed Gaussian particle field evolves under gravity, wind, and turbulence, activating latent weather priors in pretrained editors for dense, coherent particles.
• 📐 Geometry grounding — particles are gravity-aligned and projected with camera intrinsics/extrinsics into particle-augmented depth, ensuring spatially accurate, temporally consistent placement.

⭐ If you like our work or find it useful, please give us a star or cite below. Thanks!

• Jun 2026  🎉 Paper accepted to ECCV 2026!
• Jun 2026  🌐 Project page is live, with the supplementary demo video.
• Aug 2026  💻 Code Released!.
• Aug 2026  ✅ We have halved the inference time. Each clip now takes only ~20 mins.

• GPU — 48 GB recommended; synthesis peaks around 40 GB at 480×832. Measured on a single RTX A6000. On smaller cards, WEATHERCRAFTER_VACE_VRAM_LIMIT_GB offloads part of the model to CPU, trading speed for headroom.
• Disk — ~80 GB for the model weights, ideally on a fast local disk since loading is I/O-bound.
• RAM — 32 GB minimum, 64 GB comfortable.
• CUDA 12.4, Python 3.10–3.12.
• OpenAI and Black Forest Labs API keys, for the anchoring stage only.

git clone https://github.com/Jumponthemoon/WeatherCrafter.git
cd WeatherCrafter

conda env create -f environment.yml
conda activate weathercrafter

# the synthesis (VACE) stage
pip install -e ".[synthesis]"
pip install --no-build-isolation \
    "diffsynth @ git+https://github.com/modelscope/DiffSynth-Studio.git@8332ece"

# API keys
export OPENAI_API_KEY="sk-..."
export BFL_API_KEY="..."

# Download dataset
bash scripts/download_examples.sh 

# Full pipeline run
python -m weathercrafter pipeline \
    --dataset_name drone \
    --target_weather snowy \
    --appearance_stage medium \
    --particle_severity moderate

For your own clip, put it at data/<name>/<name>.mp4 and pass --dataset_name <name>. The synthesis result lands in:

output/<name>/<weather>_a_<stage>_p_<severity>/synthesis_results/result_*.mp4

--target_weather      rainy | snowy
--appearance_stage    short | medium | long      how long the weather has been going
--particle_severity   light | moderate | heavy   how much is falling right now

appearance_stage drives the first-frame edit — how far the environment has changed, from wet patches to deep accumulation. particle_severity drives the simulation — particle count, wind, and turbulence.

# Preprocessing
python -m weathercrafter preprocess --dataset_name drone

# Anchoring
python -m weathercrafter anchoring  --dataset_name drone --target_weather snowy \
    --appearance_stage medium --particle_severity moderate

# Simulation
python -m weathercrafter simulation --dataset_name drone --target_weather snowy \
    --appearance_stage medium --particle_severity moderate

# Synthesis
python -m weathercrafter synthesis  --dataset_name drone --target_weather snowy \
    --appearance_stage medium --particle_severity moderate

If you find our work useful, please consider citing:

@inproceedings{qian2026weathervid,
  title     = {Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis},
  author    = {Qian, Chenghao and Savov, Nedko and Kong, Lingdong and Jin, Yeying and
               Song, Rui and Li, Wenjing and Zhong, Zhun and Ma, Jiaqi and
               Markkula, Gustav and Van Gool, Luc},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}