Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priors

Soumava Paul, Prakhar Kaushik, Alan Yuille

Johns Hopkins University

TMLR 2025

arXiv

Paper

Code

Checkpoint

Finetuning Data

TLDR: Gaussian Scenes is a generative approach for pose-free reconstruction of 360° scenes from a limited number of uncalibrated 2D images. We train a RGBD diffusion model capable of inpainting missing content and removing artifacts from novel view renders and depth maps of a 3DGS representation fitted to sparse inputs.

Our key contributions include a pixel-aligned confidence measure for better detection of empty regions and artifacts in novel views. We also propose context and geometry conditioning through FiLM modulation layers as a lightweight alternative to cross-attention layers.

Gaussian Scenes Overview

Our model comprises a variational autoencoder operating in a compressed latent space and a UNet denoiser for predicting noise in diffused latents. The UNet receives multimodal conditioning through four inputs: an RGBD image with artifacts, a confidence map identifying unreliable regions, CLIP features of source images providing semantic context, and camera encodings capturing geometric relationships between input views.

Method	Pose-free	Open-source	Generative Priors	Scene Reconstruction
FreeNeRF, DietNeRF, RegNeRF, DN-Gaussian, SparseGS, SparseNeRF	❌	✅	❌	✅
DiffusioNeRF, ZeroNVS	❌	✅	✅	✅
ReconFusion, CAT3D	❌	❌	✅	✅
Gaussian Object, iFusion, UpFusion	✅	✅	✅	❌
InstantSplat, COGS	✅	✅	❌	✅
Gaussian Scenes (Ours)	✅	✅	✅	✅

Table: Comparison of sparse-view reconstruction methods. Methods are grouped based on their requirement for accurate camera poses, open-source availability, need for generative priors, and applicability to large-scale scene reconstruction.

Sample Scene Reconstruction

More visualizations coming soon!

MASt3R + 3DGS

Ours

Qualitative Comparison with ReconFusion and CAT3D

We compare our approach with current state-of-the-art posed reconstruction techniques in ReconFusion and CAT3D. Unfortunately, both methods do not have open-source code available. Hence, we pick the relevant test views for 4 scenes showcased in their paper - Treehill, Flowers, Bicycle from MipNeRF360, and the plant scene from CO3Dv2 for a qualitative comparison. We use the same training views as open-sourced in their data splits.

	MASt3R + 3DGS	ReconFusion	CAT3D	Ours	Ground Truth
Treehill (3)
Flowers (3)
Plant (3)
Bicycle (9)

Despite being a pose-free pipeline, our method achieves competitive novel-view synthesis (NVS) quality with state-of-the-art sparse-view reconstruction techniques. No image is available for CAT3D in the last row, hence it is left blank.

BibTeX

@article{
      paul2025gaussian,
      title={Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priors},
      author={Soumava Paul and Prakhar Kaushik and Alan Yuille},
      journal={Transactions on Machine Learning Research},
      issn={2835-8856},
      year={2025},
      url={https://openreview.net/forum?id=yp1CYo6R0r},
      note={}
      }