WorldBench Team
Manuscript and collection updated: October 16, 2026.
World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit.
While prior work largely emphasizes generative methods for 2D image and video data, they overlook the rapidly growing body of work that leverages native 3D and 4D representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds for large-scale scene modeling.
This survey reviews 3D and 4D world models — models that learn, predict, and simulate the geometry and dynamics of real environments from multi-modal signals.
We unify terminology, scope, and evaluation, and organize the space into three complementary paradigms by representation: VideoGen (image/video-centric), OccGen (occupancy-centric), and LiDARGen (point-cloud-centric).
What it is. An image/video-centric world model that generates or predicts photorealistic, temporally consistent frames (single- or multi-view) conditioned on past context and optional controls.
Typical inputs: past frames, poses/camera intrinsics-extrinsics, text/route/action hints.
What it is. An occupancy-centric world model that represents scenes as 3D/4D occupancy fields (e.g., voxel grids with semantics), enabling geometry-aware perception, forecasting, and simulation.
Typical inputs: multi-sensor cues (RGB, depth, events, LiDAR), ego motion, maps.
What it is. A point-cloud-centric world model that learns high-fidelity geometry and dynamics directly from LiDAR sweeps, suitable for robust 3D understanding, data generation, and sensor-faithful simulation.
Typical inputs: past LiDAR sweeps, ego trajectory, calibration; optional scene priors.
A Hierarchical TaxonomyWe establish precise definitions, introduce a structured taxonomy spanning VideoGen, OccGen, and LiDARGen approaches, and systematically summarize datasets and evaluation metrics tailored to 3D/4D settings.
Together, these models provide the foundation for simulation, planning, and embodied intelligence in complex environments.
World Model CategorizationWe categorize 3D/4D world models into four functional types. They differ in how historical observations are leveraged, which conditioning signals are provided, and whether models operate in open-loop or closed-loop settings.
Generate diverse 3D/4D scenes from geometric and semantic cues, optionally with action conditions. Focus on plausibility and diversity for large-scale data augmentation and scenario creation.
Forecast future 3D/4D world states from historical observations under given action conditions. Enable action-aware forecasting for planning, behavior prediction, and policy evaluation.
Iteratively simulate closed-loop agent-environment interactions by generating successive scene states. Support interactive simulation for autonomous driving, robotics, and immersive XR training.
Recover complete and coherent 3D/4D scenes from partial, sparse, or corrupted observations. Facilitate interactive tasks such as high-fidelity mapping and digital twin restoration.
Together, these four categories outline the functional landscape of 3D/4D world modeling. While all aim to produce physically and semantically coherent scenes, they differ in how they leverage past observations, conditioning signals, and interaction loops—supporting applications from large-scale data synthesis and policy evaluation to interactive simulation and scene restoration.
VideoGen
OccGen
LiDARGenIf this survey is useful to your research, please cite:
@article{survey_3d_4d_world_models,
title = {{3D} and {4D} World Modeling: A Survey},
author = {Lingdong Kong and Yu Yang and Jianbiao Mei and Youquan Liu and Ao Liang and Dekai Zhu and Dongyue Lu and Wei Yin and Xiaotao Hu and Mingkai Jia and Junyuan Deng and Kaiwen Zhang and Yang Wu and Tianyi Yan and Shenyuan Gao and Song Wang and Linfeng Li and Liang Pan and Yong Liu and Jianke Zhu and Wei Tsang Ooi and Steven C. H. Hoi and Ziwei Liu},
journal = {arXiv preprint arXiv:2509.07996},
year = {2025}
}Controlled exploration and revisits distinguish preservation of observed regions, persistence of newly generated content, and 3D self-consistency. Short and minute-long protocols combine reference checks with cross-view geometry and held-out reconstruction, while measuring camera control separately.
Visual-motion supervision and a compact metric residual connect future observations during training to precise driving trajectories. A one-step action expert predicts both components from current inputs without generating future images at inference.
Five complementary aspects connect video generation to 4D reconstruction, action-following, downstream perception, and human preference. Open- and closed-loop driving tests complement geometric checks and a learned evaluator trained on human scores and rationales.
