Kepler4D: Controllable Future Video Generation
via 4D Scene State Evolution

Feiran Wang¹, Bin Duan², Junyi Wu¹, Gaowen Liu³, Yan Yan^1,†

¹ University of Illinois at Chicago   ² University of Michigan Ann Arbor   ³ Cisco
^† Corresponding author

Observe motion. Evolve scene state. Control the future.

This is the official project repository for Kepler4D.

Kepler4D generates future videos through explicit 4D scene state evolution. Given an observed monocular video, it organizes background geometry, object motion histories, coarse spatial supports, and semantic context in a shared 3D representation. Chain-of-Motion reasons over observed motion and scene context to produce structured motion decisions, which a deterministic rollout converts into editable future object trajectories. Rendered geometric controls then guide a pretrained video generator to synthesize future frames.

Key Features:

• Explicit 4D scene state: Connect scene geometry, observed object motion, and semantic context in a shared world frame.
• Chain-of-Motion: Turn structured speed and heading decisions into inspectable future 3D trajectories through deterministic rollout.
• Language-guided trajectory editing: Specify alternative object motion before video synthesis.
• Viewpoint-aware generation: Render the scene state and object trajectories under target camera paths to guide future video generation.

1. Construct the scene state. Recover background geometry and object proxies with observed 3D trajectories, spatial supports, and semantic context.
2. Evolve object motion. Infer structured motion decisions and expand them into future 3D trajectories that can be inspected or edited.
3. Generate future video. Render depth maps, object control signals, and masks to guide a pretrained video generator. The trajectories specify coarse object motion, while the generator synthesizes appearance and local articulation.

High-level instructions such as "Go Straight," "Turn Left 60°," and "Turn Left 90°" produce different object trajectories from the same observed video.

The second column shows the inferred trajectories. The comparison methods receive the same language commands; Kepler4D additionally uses its inferred trajectories and rendered geometric controls.

If you find this work useful, please consider citing it:

@misc{wang2026kepler4d,
  title={{Kepler4D}: Controllable Future Video Generation via 4D Scene State Evolution},
  author={Wang, Feiran and Duan, Bin and Wu, Junyi and Liu, Gaowen and Yan, Yan},
  year={2026},
  url={https://github.com/Brack-Wang/kepler4d}
}