[ECCV 2026] Learning Video Dynamics with Predictive Differentiable Rendering

The official implementation of the ECCV'26 paper Learning Video Dynamics with Predictive Differentiable Rendering.

Yujin Tang*, Tian Zhou*, Xin Lin, Cheng Tan, Yifan Hu, Rong Jin, Souyong Jin, Liang Sun.

European Conference on Computer Vision (ECCV), 2026.

PredGS augments spatiotemporal predictive models with a Predictive Differentiable Rendering (PDR) adapter: future frames are rendered in high fidelity from a set of learned 2D Gaussians via differentiable 2D Gaussian splatting. The renderer is implemented as a custom CUDA extension (predgs/) with analytic forward/backward passes. The training framework is built on OpenSTL.

PredGS
├── configs/            # per-dataset training configs (PredGS_* and baselines)
├── openstl/            # OpenSTL-based framework (methods / models / modules / datasets)
│   └── modules/predgs_modules.py   # Python wrapper of the CUDA renderer
├── predgs/             # the 2D Gaussian splatting CUDA extension
├── scripts/            # example train / test shell scripts per dataset
└── tools/              # train.py / test.py entry points, data download scripts

The PredGS method (method key predgs) uses a SimVP-family backbone (model_type: 'tau' or 'gSTA' in the config). Example configs: configs/taxibj/PredGS_SimVP.py, configs/kth/PredGS_SimVP.py, configs/human/PredGS_SimVP.py, configs/sevir/PredGS_SimVP.py, configs/weather/t2m_5_625/PredGS_SimVP.py.

Tested with Python 3.10, PyTorch 2.1.0 (CUDA 12.1 wheels), and CUDA toolkit 12.x for compiling the extension.

# 1. create the environment
conda create -n predgs python=3.10 -y
conda activate predgs

# 2. install PyTorch (pick the wheel matching your CUDA setup)
pip install torch==2.1.0 torchvision==0.16.0

# 3. install the framework
pip install -r requirements.txt
pip install -e .

# 4. compile the PredGS CUDA extension (requires nvcc from CUDA toolkit 12.x)
export TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;9.0"   # adjust to your GPUs
cd predgs
pip install --no-build-isolation .
cd ..

# 5. sanity check
python -c "import openstl, predgs; print('ok')"

Notes:

• TORCH_CUDA_ARCH_LIST controls which GPU architectures the extension is compiled for (7.0 V100, 8.0 A100, 8.6 A5000/A5500/RTX30xx, 9.0 H100/H200).
• If nvcc is not on your PATH, set CUDA_HOME to your CUDA toolkit location (e.g. export CUDA_HOME=/usr/local/cuda-12.1).

On clusters where the system CUDA is provided via environment modules and PyTorch is installed from pip wheels, you may hit libnvJitLink version errors (undefined symbol: __nvJitLinkAddData_12_1) when importing torch — including inside pip install --no-build-isolation ., whose setup.py imports torch. Fix (set these before compiling or running anything):

module load cuda/12                      # or your site's CUDA 12 module
export CUDA_HOME=/path/to/cuda/12
pip install --upgrade nvidia-nvjitlink-cu12
# pip's nvjitlink must come FIRST so it shadows the older system libnvJitLink
export LD_LIBRARY_PATH=$(python -c "import nvidia.nvjitlink; print(nvidia.nvjitlink.__path__[0])")/lib:$CUDA_HOME/lib64:$LD_LIBRARY_PATH

If compiling the extension fails with ModuleNotFoundError: No module named 'pkg_resources', your setuptools is too new for torch 2.1's cpp_extension; run pip install "setuptools<81" (already pinned in requirements.txt).

If you recompile for a different GPU, clear stale build artifacts first: rm -rf predgs/build predgs/*.egg-info.

Datasets are expected under ./data. Download scripts are provided in tools/prepare_data/, e.g. for TaxiBJ (used in the quick start below):

bash tools/prepare_data/download_taxibj.sh

KTH: bash tools/prepare_data/download_kth.sh; Human3.6M: bash tools/prepare_data/download_human3.6m.sh; WeatherBench: bash tools/prepare_data/download_weatherbench.sh (also pip install xarray netcdf4).

data/
├── taxibj/dataset.npz
├── kth/...
└── ...

# TaxiBJ, PredGS with the SimVP/TAU backbone
bash scripts/taxibj/taxibj_PredGS_SimVP.sh

# or call the entry point directly
python tools/train.py \
    --config_file configs/taxibj/PredGS_SimVP.py \
    --dataname taxibj \
    --data_root data \
    --res_dir work_dirs \
    --batch_size 16 --epoch 50 --lr 5e-4 --alpha 0.5 \
    --opt adamw --weight_decay 1e-2 \
    --ex_name taxibj/PredGS_SimVP

Results (checkpoints, logs, TensorBoard scalars) are written to work_dirs/<ex_name>. Example scripts for KTH / Human3.6M / SEVIR / WeatherBench are in scripts/<dataset>/.

python tools/test.py \
    --config_file configs/taxibj/PredGS_SimVP.py \
    --dataname taxibj \
    --data_root data \
    --test \
    --ex_name work_dirs/taxibj/PredGS_SimVP

@inproceedings{tang2026predgs,
  title     = {Learning Video Dynamics with Predictive Differentiable Rendering},
  author    = {Tang, Yujin and Zhou, Tian and Lin, Xin and Tan, Cheng and Hu, Yifan and Jin, Rong and Jin, Souyong and Sun, Liang},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

This codebase is built on OpenSTL (Apache-2.0). We thank the authors for their excellent framework.

Released under the Apache-2.0 license. See LICENSE.