BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering

BRDFusion combines physics-based inverse rendering with generative modeling for high-quality urban scene relighting. It decomposes urban videos into geometry, materials, and HDR lighting for novel-view rendering, relighting, and scene-edit applications.

• 🚀 Quick Start
  • 🔦 Prerequisites
  • ⚙️ Installation
  • 📦 Download Data
  • 🧩 Download Pretrained Checkpoints
  • 🖼️ Render from a Checkpoint
  • ✨ Gen. Render
  • 📊 Metric Computation
  • 🎬 Applications
• 🏋️ Training
• 🛣️ Additional Waymo Scenes Processing

Use the provided preprocessed datasets and pretrained checkpoints for direct inference and evaluation. Training is covered in Training, and optional Waymo preprocessing is covered in Additional Waymo Scenes Processing.

The codebase has been tested on:

• OS: Ubuntu 22.04
• GPU: NVIDIA RTX A6000
• C compiler: gcc and g++ version 11. We assume that the system provides the aliases gcc-11 and g++-11.
• Memory note: An RTX 4090 can also run some stages when memory usage allows, but the Gen. Render stage can exceed the 24 GB memory limit.

BRDFusion uses two environments for the main workflows:

Note that we install CUDA 11.8 in the brdfusion env, while using CUDA 12.1 in the system for cosmos-predict1. If your system don't have CUDA 12.1 installed, install it in cosmos-predict1 as well.

Clone the repository:

git clone --recursive https://github.com/shigon255/BRDFusion.git
cd BRDFusion

conda create -n brdfusion python=3.11
conda activate brdfusion

cd third_party/3dgrut
conda install -y cuda-toolkit cmake ninja -c nvidia/label/cuda-11.8.0
conda install -y pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 "numpy<2.0" "mkl=2023.2.0" -c pytorch -c nvidia/label/cuda-11.8.0 -c conda-forge
CC=gcc-11 CXX=g++-11 pip install --find-links https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.1.2_cu118.html kaolin==0.17.0
conda install -c conda-forge mesa-libgl-devel-cos7-x86_64 -y
pip install setuptools==71.1.0

git submodule update --init --recursive
CC=gcc-11 CXX=g++-11 python -m pip install --no-build-isolation -r requirements.txt
CC=gcc-11 CXX=g++-11 pip install -e .

cd ../../
CC=gcc-11 CXX=g++-11 pip install --no-build-isolation -r requirements.txt
CC=gcc-11 CXX=g++-11 python -m pip install --no-build-isolation git+https://github.com/nerfstudio-project/gsplat.git@v1.3.0
CC=gcc-11 CXX=g++-11 pip install --no-build-isolation git+https://github.com/facebookresearch/pytorch3d.git
CC=gcc-11 CXX=g++-11 pip install --no-build-isolation git+https://github.com/NVlabs/nvdiffrast
cd third_party/smplx
pip install -e .
cd ../..
pip install numpy==1.26.4
conda deactivate

cd third_party/cosmos1-diffusion-renderer
conda env create --file cosmos-predict1.yaml
conda activate cosmos-predict1
pip install -r requirements.txt

ln -sf $CONDA_PREFIX/lib/python3.10/site-packages/nvidia/*/include/* $CONDA_PREFIX/include/
ln -sf $CONDA_PREFIX/lib/python3.10/site-packages/nvidia/*/include/* $CONDA_PREFIX/include/python3.10
pip install transformer-engine[pytorch]==1.12.0

ln -sf $CONDA_PREFIX/lib/python3.10/site-packages/triton/backends/nvidia/include/crt $CONDA_PREFIX/include/
CC=gcc-11 CXX=g++-11 pip install --no-build-isolation git+https://github.com/NVlabs/nvdiffrast.git

CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) \
  python scripts/download_diffusion_renderer_checkpoints.py --checkpoint_dir checkpoints
cd ../..
conda deactivate

Prepare SMPL assets:

1. Download SMPL v1.1 (SMPL_python_v.1.1.0.zip) from the SMPL official website.
2. Move SMPL_python_v.1.1.0/smpl/models/basicmodel_neutral_lbs_10_207_0_v1.1.0.pkl to smpl_models/SMPL_NEUTRAL.pkl.

We provide preprocessed evaluation data for selected Waymo Open Dataset scenes and all synthetic dataset scenes.

The synthetic dataset capture the urban scene in Blender using 3 cameras. We render 6 paths in the scene. Each path includes 100 ground-truth RGB, material, and relighting RGB frames. For each path we render a shifted version (path*-3-calib_*) of it used for the reported NVS, inverse rendering, and NVS-relight metrics. Note that we use 1 camera and the first 51 frames for evaluation. The synthetic 3D assets come from BlenderKit, and the HDRIs come from PolyHaven.

Download the preprocessed datasets from huggingface, unzip them under the repository root, and keep this top-level layout. Note that training, rendering, and computation for each scene only require the data for that specific scene. Since the full dataset is quite large, you can download only the subset of scenes you want to try.

data/
  self/
    path{1..6}_fixed_tree_gamma_full/
    path{1..6}-3-calib_fixed_tree_gamma_full/
  waymo/processed/training/{003,019,114,172,703}/

data/self/path1_fixed_tree_gamma_full/qwantani_moon_noon_puresky_4k/
  Camera_Center_poses.txt
  Cam_Left_poses.txt
  Cam_Right_poses.txt
  images/
  intrinsics/
  sky_masks/
  gt_sky_mask/
  depth/
  normal/
  albedo/
  roughness/
  metallic/
  diffusion_renderer_depth/
  diffusion_renderer_normal/
  diffusion_renderer_albedo/
  diffusion_renderer_roughness/
  diffusion_renderer_metallic/
  dlenvmap/
  qwantani_moon_noon_puresky_4k.exr

data/self/path1_fixed_tree_gamma_full/qwantani_moon_noon_puresky_4k_rot90/
  Camera_Center_poses.txt
  Cam_Left_poses.txt
  Cam_Right_poses.txt
  images/
  intrinsics/
  qwantani_moon_noon_puresky_4k_rot90.exr
  qwantani_moon_noon_puresky_4k_rot90.hdr

data/self/path1-3-calib_fixed_tree_gamma_full/qwantani_moon_noon_puresky_4k/
  Camera_Center_poses.txt
  Cam_Left_poses.txt
  Cam_Right_poses.txt
  images/
  intrinsics/
  gt_sky_mask/
  depth/
  normal/
  albedo/
  roughness/
  metallic/
  colmap_1cam/
  qwantani_moon_noon_puresky_4k.exr

data/waymo/processed/training/003/
  images/
  lidar/
  ego_pose/
  extrinsics/
  intrinsics/
  sky_masks/
  dynamic_masks/
  instances/
  diffusion_renderer_depth/
  diffusion_renderer_normal/
  diffusion_renderer_albedo/
  diffusion_renderer_roughness/
  diffusion_renderer_metallic/
  dlenvmap/

For additional Waymo scenes, see Additional Waymo Scenes Processing.

We provide 1-camera checkpoints trained on frames 0..50 with test_image_stride=10. Download the checkpoints from huggingface and unzip them under ckpt/:

ckpt/
  self/path{1..6}_fixed_tree_gamma_full/qwantani_moon_noon_puresky_4k/checkpoint_final.pth
  waymo/{003,019,114,172,703}/checkpoint_final.pth

Stage the downloaded checkpoints into the run-folder layout used by the rendering tools:

bash scripts/stage_ckpt.sh

The staging script places checkpoints under the corresponding work_dirs/ run directories for tools/run_pipeline.py and scripts/applications/render.sh.

After staging checkpoints, PBR-render through tools/run_pipeline.py. The common targets are:

Waymo example:

python tools/run_pipeline.py \
  --dataset waymo \
  --cams 1 \
  --scene_idx 3 \
  --stage render \
  --render_target recon

Synthetic dataset example:

python tools/run_pipeline.py \
  --dataset self \
  --cams 1 \
  --path_id 1 \
  --stage render \
  --render_target all \
  --relight_scene_idx qwantani_moon_noon_puresky_4k_rot90

Gen. Render runs DiffusionRenderer forward refinement on videos produced by the render stage. Use the same dataset, scene/path, camera setting, frame range, and render targets as the render command.

Waymo:

python tools/run_pipeline.py \
  --dataset waymo \
  --cams 1 \
  --scene_idx 3 \
  --stage gen_render \
  --render_target recon

Synthetic dataset:

python tools/run_pipeline.py \
  --dataset self \
  --cams 1 \
  --path_id 1 \
  --stage gen_render \
  --render_target all

Gen. Render outputs are written next to the rendered videos for the selected target.

Metric computation assumes Gen. Render has completed for the selected targets. Waymo computes NVS image metrics. Synthetic dataset scenes compute NVS, relighting, and intrinsic metrics on both original and shifted paths.

Waymo:

python tools/run_pipeline.py \
  --dataset waymo \
  --cams 1 \
  --scene_idx 3 \
  --stage compute_metrics \
  --render_target recon

Synthetic dataset:

python tools/run_pipeline.py \
  --dataset self \
  --cams 1 \
  --path_id 1 \
  --stage compute_metrics \
  --render_target all

Render outputs are written under render/<target>/, and metric JSONs are written under matching metrics/<target>/ folders.

For direct comparison with BRDFusion and baselines such as UrbanIR, InvRGB+L, and Gen3C+DR, download the precomputed videos from huggingface and unzip them under videos/:

videos/
  brdfusion/{self,waymo}/...
  gen3c_dr/{self,waymo}/...
  invrgbl/{self,waymo}/...
  urbanir/{self,waymo}/...

Compute metrics on the downloaded videos with:

conda run -n brdfusion --no-capture-output \
  python -u tools/metrics/compute_precomputed_video_metrics.py --root ./videos/

The application renderer applies relighting, camera-path changes, local lights, headlights, and dynamic object edits at evaluation time without changing the checkpoint.

CKPT=/path/to/checkpoint_final.pth \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
NEW_ENVMAP=/path/to/target_envmap.hdr \
NEW_ENVMAP_RES=1024 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
START=10 END=30 \
CAM_IDS=0 \
CAMERA_INTERP_STEPS=2 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
TIMESTEP=20 CAM_ID=0 \
SPIRAL_TRAJECTORY_MODE=circle \
SPIRAL_CIRCLE_START_SIDE=bottom_left \
SPIRAL_CIRCLE_RADIUS_RIGHT_M=0.5 \
SPIRAL_CIRCLE_RADIUS_UP_M=0.3 \
SPIRAL_FRAMES=120 \
SPIRAL_LOOPS=1 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
TIMESTEP=20 CAM_ID=0 \
SPIRAL_TRAJECTORY_MODE=spiral \
SPIRAL_FRAMES=120 \
SPIRAL_LOOPS=1 \
SPIRAL_RADIUS_M=1.0 \
SPIRAL_VERTICAL_AMPLITUDE_M=0.3 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
NEW_ENVMAP=/path/to/NightSkyHDRI001_8K_HDR.exr \
NEW_ENVMAP_RES=4096 \
LOCAL_LIGHT_CONFIG=configs/application/local_light_example.json \
POINT_LIGHTS_USE_ENVMAP=1 \
RENDER_POINT_LIGHT_EMITTERS=1 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
NEW_ENVMAP=/path/to/NightSkyHDRI001_8K_HDR.exr \
NEW_ENVMAP_RES=4096 \
POINT_LIGHTS_FILE=assets/blender_light_pos.txt \
POINT_LIGHTS_FILE_FORMAT=blender_area_txt \
POINT_LIGHTS_FILE_ENERGY_SCALE=1.0 \
RENDER_POINT_LIGHT_EMITTERS=1 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
NEW_ENVMAP=/path/to/NightSkyHDRI001_8K_HDR.exr \
NEW_ENVMAP_RES=4096 \
HEADLIGHTS=1 \
HEADLIGHT_INTENSITY=150 \
HEADLIGHT_LATERAL_OFFSET=0.8 \
HEADLIGHT_DOWN_OFFSET=0.3 \
HEADLIGHT_FORWARD_OFFSET=1.5 \
HEADLIGHT_AS_SPOTLIGHT=1 \
HEADLIGHT_INNER_ANGLE_DEG=40 \
HEADLIGHT_OUTER_ANGLE_DEG=60 \
scripts/applications/render.sh

conda run -n brdfusion --no-capture-output \
SOURCE_CKPT=/path/to/source/checkpoint_final.pth \
OUTPUT_PATH=/path/to/exported_assets \
EXPORT_MODE=per_object \
scripts/assets/export_dynamic_assets.sh

CKPT=/path/to/checkpoint_final.pth \
INSERT_ASSET=/path/to/exported_object.pth \
INSERT_ANCHOR_TIMESTEP=30 \
INSERT_ANCHOR_CAM_ID=0 \
INSERT_FORWARD_M=10.0 \
INSERT_RIGHT_M=-2.0 \
INSERT_UP_M=0.0 \
INSERT_YAW_DEG=0 \
INSERT_SCALE=1.0 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
INSERT_SPEC=configs/application/scene_edit_example.json \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
MOVE_SPEC=configs/application/object_move_example.json \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
MOVE_RIGID_IDS=0 \
MOVE_ANCHOR_TIMESTEP=30 \
MOVE_FORWARD_M=0.0 \
MOVE_RIGHT_M=2.0 \
MOVE_UP_M=0.0 \
MOVE_YAW_DEG=15 \
MOVE_SCALE=1.0 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
DELETE_RIGID_IDS=3,5 \
DELETE_SMPL_IDS=0 \
scripts/applications/render.sh

CKPT=/path/to/checkpoint_final.pth \
NO_PBR=1 \
PRINT_CMD=1 \
scripts/applications/render.sh

VIDEO_ROOT=/path/to/application/videos \
STRENGTHS=0.5 \
scripts/render/gen_render_video_folder.sh

Training can be run on the provided synthetic dataset or on prepared Waymo scenes. The same driver handles training, rendering, Gen. Render, and metric computation.

Use tools/run_pipeline.py as the main entrypoint. By default, it runs the full staged pipeline; --stage lets you run one part of the workflow.

The released checkpoints use frames 0..50 with test_image_stride=10, and the examples below follow the same setting.

The pipeline has four user-facing stages:

Train on the provided synthetic dataset:

python tools/run_pipeline.py \
  --dataset self \
  --cams 1 \
  --path_id 1 \
  --start_timestep 0 \
  --end_timestep 50 \
  --test_image_stride 10

Train on a prepared Waymo scene:

python tools/run_pipeline.py \
  --dataset waymo \
  --cams 1 \
  --scene_idx 3 \
  --start_timestep 0 \
  --end_timestep 50 \
  --test_image_stride 10

Use --cams 3 for the 3-camera setting. Add --dry_run to print the selected commands and write pipeline_manifest.json without launching heavy jobs.

Run a single stage when moving expensive work between machines:

python tools/run_pipeline.py \
  --dataset self \
  --cams 1 \
  --path_id 1 \
  --start_timestep 0 \
  --end_timestep 50 \
  --test_image_stride 10 \
  --stage gen_render

When executing, stages whose completion outputs already exist are skipped by default. For example, if stage1a/checkpoint_final.pth exists, --stage train continues from Gen. Refine. Add --rerun_existing to force recomputation.

Processing additional Waymo scenes is optional. Use this section when you want to train or evaluate BRDFusion on scenes beyond the provided preprocessed Waymo data.

data/waymo/processed/training/
  003/
    images/
    lidar/
    ego_pose/
    extrinsics/
    intrinsics/
    sky_masks/
    dynamic_masks/

DATA_ROOT=data/waymo/processed/training \
SCENE=3 \
CAM_IDS="0" \
NUM_TIMESTEPS=51 \
scripts/priors/run_dr_waymo.sh

cd third_party/DiffusionLight-Turbo
conda env create -f environment.yml
conda activate diffusionlight
pip install -r requirements.txt
cd ../../

DATA_ROOT=data/waymo/processed/training \
SCENE=3 \
INTERVAL=2 \
CAM_IDS="0" \
NUM_TIMESTEPS=51 \
scripts/priors/run_dl_waymo.sh

DATA_ROOT=data/waymo/processed/training \
SCENE=3 \
INTERVAL=2 \
CAM_IDS="0" \
NUM_TIMESTEPS=51 \
scripts/priors/merge_dl_waymo.sh

This project builds on DriveStudio, 3DGRUT, DiffusionRenderer, and DiffusionLight-Turbo. We thank the authors of these projects for releasing their code and resources.

This research was funded by the National Science and Technology Council, Taiwan, under Grants NSTC 112-2222-E-A49-004-MY2 and 113-2628-E-A49-023-. The authors are grateful to Google, NVIDIA, and MediaTek Inc. for their generous donations. Yu-Lun Liu acknowledges the Yushan Young Fellow Program by the MOE in Taiwan.

If you find BRDFusion useful for your research, please consider citing:

@misc{liu2026brdfusionphysicsmeetsgeneration,
      title={BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering}, 
      author={Yi-Ruei Liu and Jie-Ying Lee and Zheng-Hui Huang and Yu-Lun Liu and Chih-Hao Lin},
      year={2026},
      eprint={2606.17049},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.17049}, 
}

BRDFusion source code is released under the MIT license in LICENSE. Third-party components under third_party/ are governed by their respective licenses.