Pandora: Towards General World Model
with Natural Language Actions and Video States
We introduce Pandora, a step towards a General World Model (GWM) that:
- Simulates world states by generating videos across any domains
- Allows any-time control with actions expressed in natural language
On-the-Fly Control with Natural Language
Pandora accepts free-text actions as inputs during video generation to steer the video on the fly. This differs crucially from previous text-to-video models which only allow text prompts at the beginning of the video. The on-the-fly control fulfills the promise of the world model to support interactive content generation and enhance robust reasoning and planning.
Predicting Alternative Futures as You Like
World models simulate alternative futures of the world. Pandora allows you to control the future. Here we show some counterfactual futures – different videos generated from the same initial state but different actions.
Initial state
Future 1
Future 2
Action: Magma erupts from the crater
Action: The sky gets dark
Action: Turn left. There is a white van
Action: Turn right. There is a red car
Action: The man waves his hand
Action: Two men walks towards the microphone
Action: Use spoon to scoop some broccoli
Action: Use spoon to stir the rice
Simulating Worlds across Any Domains
Pandora is capable of generating videos across a wide range of general domains, such as indoor/outdoor, natural/urban, human/robot, 2D/3D, and other scenarios. You may find more videos in the Pandora’s Box gallery.
Action: Pouring milk into the glass cup from a milk bottle
Action: Flame ignites woods emitting some smoke
Action: Wind blows the leaves
Action: Let the traffic go
Action: The man drops the bag
Action: Fold the towel
Action: Jump right
Action: Look around
Action: Set fire on the river
Action: Fireworks bloom in the night sky
Learning Actions in One Domain and Using in Another
Instruction tuning with high-quality data allows the model to learn effective action control and transfer to different unseen domains. E.g., Pandora saw the only 2D game Coinrun during training, but can seamlessly apply the learned actions to other 2D games.
Source Domain
Target Domain
Source Domain
Target Domain
Autoregressive Model Yields Longer Videos
Existing diffusion video models typically produce videos of a fixed length. By integrating the video model with the Pandora autoregressive backbone, longer videos with potentially unlimited duration can be generated. We show 8-second videos generated by Pandora, even though our training videos are up to 5 seconds.
Action: The car moves forward
Action: The man is flying
Action: Walk into the theatre
Action: The car moves forward
Action: Let the train move
Action: The plane is flying
Limitation
Pandora as a prelimiary step towards GWM is still limited. It can fail in generating consistent videos, simulating complex scenarios, understanding commonsense and physical laws, and following instructions/actions.
Action: Pick up the wallet
Action: Take out the nozzle
Action: The man is dancing
Action: The train door is open
Note
We processed the videos on this website with FLAVR for frame interpolation to make them smoother. No other post-processing used.
If you have concern about the copyright of any image/video on this website, please contact us.
Pandora Team
Jiannan Xiang, Guangyi Liu, Yi Gu (equal contribution)
Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu
Eric P. Xing, Zhiting Hu
Maitrix.org
UCSD
MBZUAI
1
2
3
4
5
6
@article{xiang2024pandora,
title={Pandora: Towards General World Model with Natural Language Actions and Video States},
author={Jiannan Xiang and Guangyi Liu and Yi Gu and Qiyue Gao and Yuting Ning and Yuheng Zha and Zeyu Feng and Tianhua Tao and Shibo Hao and Yemin Shi and Zhengzhong Liu and Eric P. Xing and Zhiting Hu},
journal={arXiv preprint arXiv:2406.09455},
year={2024}
}