ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectories

ReCamDriving is a LiDAR-free, camera-controlled video synthesis framework for generating driving videos along novel lateral trajectories. Given a source video, the target camera displacement, and an aligned 3D Gaussian Splatting (3DGS) rendering from the laterally shifted target trajectory, it produces target-view videos with consistent spatial structure and temporal dynamics.

Create the tested Python 3.10 environment and install the current source checkout:

git clone https://github.com/Iron-LYK/ReCamDriving.git
cd ReCamDriving
conda env create -f environment.yml
conda activate recamdriving
python -m pip install -e .

Run the commands below from the repository root.

Download the ReCamDriving checkpoint and place it at:

checkpoints/ReCamDriving.ckpt

ReCamDriving also requires Wan2.1_VAE.pth from Wan2.1-T2V-1.3B. Download the file and place it at:

checkpoints/Wan2.1_VAE.pth

The commands below pass this local path explicitly. Alternatively, if --wan-vae is omitted, inference downloads the pinned VAE revision to the Hugging Face cache, normally ~/.cache/huggingface/hub, rather than to checkpoints/. The cache location can be changed with HF_HOME or HF_HUB_CACHE.

Each inference run requires three inputs: a clean source_video from the original trajectory, a condition_video rendered from the reconstructed 3DGS along a laterally shifted target trajectory, and the signed displacement passed through --lateral-offset-m. The code constructs the corresponding relative camera pose automatically. Positive offsets move the camera right; negative offsets move it left.

The source and condition videos must cover the same time interval. The condition video is typically rendered from a 3DGS reconstruction of the scene depicted in the source video. Although our work focuses on driving scenes, the method is not inherently limited to autonomous driving and may also be applied to general scenes. ReCamDriving is also independent of the 3DGS reconstruction approach: both per-scene optimization and feed-forward reconstruction can provide the condition video.

For quick evaluation, this repository includes four per-scene optimized examples in examples/scene_1 through examples/scene_4, and four feed-forward examples in examples/gaussianstorm/scene_1 through examples/gaussianstorm/scene_4.

DriveStudio is used as the representative per-scene optimization pipeline. The provided conditions were rendered after optimizing a separate 3DGS model for each driving scene.

CUDA_VISIBLE_DEVICES=0 python infer \
  --source-video examples/scene_1/source.mp4 \
  --condition-video examples/scene_1/condition_left_3m.mp4 \
  --lateral-offset-m -3 \
  --checkpoint checkpoints/ReCamDriving.ckpt \
  --wan-vae checkpoints/Wan2.1_VAE.pth \
  --output outputs/drivestudio_scene_1_left_3m.mp4 \
  --seed 0 \
  --steps 50 \
  --device cuda:0

See the original DriveStudio repository for its scene reconstruction pipeline. Detailed scripts for rendering shifted trajectories with DriveStudio and constructing the ParaDrive training dataset will be released separately.

GaussianSTORM is used as the representative feed-forward reconstruction pipeline. It predicts scene Gaussians from image context and renders the requested target views without per-scene optimization.

CUDA_VISIBLE_DEVICES=0 python infer \
  --source-video examples/gaussianstorm/scene_2/source.mp4 \
  --condition-video examples/gaussianstorm/scene_2/condition_right_3m.mp4 \
  --lateral-offset-m 3 \
  --checkpoint checkpoints/ReCamDriving.ckpt \
  --wan-vae checkpoints/Wan2.1_VAE.pth \
  --output outputs/gaussianstorm_scene_2_right_3m.mp4 \
  --seed 0 \
  --steps 50 \
  --device cuda:0

The separate GaussianSTORM-ReCamDriving companion repository provides the complete setup, example and custom Waymo data preparation, and lateral-view rendering. Follow its README for the tested environment, data, and rendering instructions.

ReCamDriving builds on ReCamMaster, DiffSynth-Studio, and Wan2.1. Original ReCamDriving code is MIT-licensed, while bundled DiffSynth-derived code remains under Apache-2.0. See LICENSE for the applicable terms and third-party attributions.

If you find our work helpful, please consider citing:

@article{li2025recamdriving,
  title={ReCamDriving: LiDAR-Free Camera-Controlled Novel Trajectory Video Generation},
  author={Li, Yaokun and Wang, Shuaixian and Guo, Mantang and Huang, Jiehui and Ding, Taojun and Hu, Mu and Wang, Kaixuan and Shen, Shaojie and Tan, Guang},
  journal={arXiv preprint arXiv:2512.03621},
  year={2025}
}