🚧 This repository is currently under construction, and we are continually developing and refining the method. Check back soon for updates! 🚧

🚧 We will provide updated complete version of code and checkpoints upon paper acceptance 🚧

This repository contains the code and models for our paper:

PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation

PPS-Ctrl is a image translation framework that combines Stable Diffusion and ControlNet, guided by a Per-Pixel Shading (PPS) map — a physics-informed representation capturing surface-light interactions. Unlike prior sim-to-real approaches that condition on depth maps, PPS provides a more faithful and geometrically consistent structural prior, enabling better texture realism and structure preservation in endoscopy image translation.

We recommend Python 3.9 with PyTorch ≥ 2.0 and the HuggingFace diffusers library.

conda create -n ppsctrl python=3.9
conda activate ppsctrl
pip install -r requirements.txt

Download the following datasets:

SimCol3D

C3VD

Colon10K

Precompute PPS maps:

python utils/compute_pps.py --depth_dir path/to/depth --output_dir path/to/pps

bash scripts/train_sd.sh

bash scripts/train_controlnet.sh

python scripts/infer.py --depth path/to/test/depth --output path/to/save

We provide pretrained checkpoints for:

Stable Diffusion (domain-finetuned)

ControlNet with PPS-encoder-decoder

📥 Download links coming soon!

This work builds on Stable Diffusion, ControlNet, Hugging Face Diffusers and PPSNet. We thank the maintainers for their open-source contributions.