OVOW: One Video, One World

Turning Monocular Video into Physical 4D Scenes

English | 中文

OVOW is the first training-free system that reconstructs instance-level, simulation-ready 4D mesh scenes from a single monocular video (or a single image): per-object 3D meshes + poses + deformation animation, assembled into a full-scene animated GLB with rendered comparison videos. Heavy models (SAM3 / Hi3DGen / FoundationPose / RoMaV2) support resident serving: loaded once for an entire batch.

⚠️ Non-commercial research use only. OVOW's own code is MIT-licensed, but this release bundles third-party components under non-commercial licenses (NVIDIA FoundationPose, CC-BY-NC SpaTrackerV2, CC-BY-NC-SA Motion324, Meta SAM License sam3, etc.). By using it you agree to each component's license — see NOTICE and THIRD_PARTY.md. Consequently, the assembled repository is a source-available research release, not an OSI-approved open-source distribution.

Prerequisites: Linux + NVIDIA GPU (≥40GB VRAM recommended, driver ≥550) · CUDA Toolkit 12.1+ (with nvcc) · conda · ~120GB disk.

# 0) Clone
git clone https://github.com/SparcAI-Inc/OVOW.git && cd OVOW

# 1) Create the 3 conda envs (installs torch + deps + compiles CUDA extensions;
#    the slowest step, ~30-60 min)
bash environment/create_envs.sh        # prints the 3 interpreter paths to paste into .env

# 2) Machine config
cp .env.example .env                   # ① paste the 3 python paths printed above
                                       # ② set DASHSCOPE_API_KEY, or set QWEN_BACKEND="local" (offline)

# 3) Install blender (hard dependency of the assembly/rendering steps)
sudo apt install -y blender            # or the official portable build → set BLENDER_BIN in .env

# 4) Localize paths (expands the repo's placeholders with your .env; one-shot —
#    to change .env later, `git checkout` the tree first, then rerun)
bash setup.sh

# 5) Download weights (~55GB, fully automatic; SAM3 is a gated HF repo —
#    `huggingface-cli login` and accept its terms on the web page first)
bash download_ckpts.sh

# 6) Reproduce the bundled examples (see below)
bash examples/smoke_test.sh          # single image, short chain, ~3-5 min
bash examples/smoke_test_video.sh    # single video, full 4D chain, ~19-22 min

If a step fails, see Troubleshooting below. For the per-environment torch/CUDA version table and the manual, step-by-step install commands that create_envs.sh automates, see environment/README.md.

The reproduction target is the bundled examples/ — running both of them through to their artifacts is the bar for a working installation. Both scripts start with a preflight that names exactly what is missing (Qwen backend / blender / conda interpreters / SAM3 and Motion324 weights) instead of failing halfway through, and both end by asserting that the key artifacts exist and are non-empty — a zero exit code alone does not count as reproduced.

Both assert these three artifacts (.png for the image chain, .mp4 for the video chain):

scene_anim_blender_physics_with_deform.glb                          # final full-scene GLB
fullscene_deform_objseq_white_left_rgb_right_split10_h264.{png,mp4} # white-model vs input
fullscene_deform_objseq_textured_left_rgb_right_split10_h264.{png,mp4} # textured vs input

smoke_test_video.sh also takes a video path, so you can point it at your own clip: bash examples/smoke_test_video.sh /path/to/your.mp4.

Note that the generative steps do not run with a fixed seed, so meshes and renders differ between runs. The bar is that the chain completes and produces the artifacts above, not bit-identical output.

# A single image
bash scripts/run_image.sh --image /path/x.png --out ./outputs

# A single video
bash scripts/run_video.sh --name eagle --video /path/eagle.mp4 --out ./outputs

# All videos in a folder (resident serving: models loaded once for the whole batch) — recommended at scale
bash scripts/run_video_batch.sh --video-dir /path/videos --gpu 0 --out ./outputs

# Mixed images+videos, multi-GPU, resumable (100k-scale)
bash scripts/run_batch.sh --in /path/inputs --gpus 0,1,2,3 --out ./outputs

Outputs: videos under <out>/ws/<name>_<timestamp>_<key>/, single images under <out>/image_pipeline_<timestamp>/<name>/:

• scene_anim_blender_physics_with_deform.glb — final full-scene animated GLB
• fullscene_deform_*_h264.mp4 — white-model / textured rendered comparison videos

benchmark/ holds OVOW-Bench, the evaluation toolkit used in the paper. It scores already-reconstructed scenes and reports Scene-IoU (AABB and OBB), Object-IoU after Hungarian matching with an object-count penalty, a photometric loss, and an N-CLIP visual metric. Geometry is compared after the alignment the protocol specifies; visual metrics use GT-camera renders.

The toolkit ships evaluator code and protocol metadata only — no ground truth, results, or predictions. The data (OVOW_Bench_Data) is published separately on Hugging Face. It needs its own conda environment plus Blender and a local CLIP checkpoint, so it is deliberately kept apart from the three pipeline envs:

conda env create -f benchmark/environment.yml && conda activate ovow-bench
python benchmark/scripts/check_env.py

# unit tests need neither Blender nor CLIP weights
cd benchmark && PYTHONDONTWRITEBYTECODE=1 python -m unittest discover -s tests

Protocol, submission layout, and the full evaluator invocation are documented in benchmark/README.md and benchmark/docs/EVALUATION_PROTOCOL.md.

The entry scripts accept an explicit list of step numbers as trailing arguments (bash scripts/run_video.sh --name eagle --video x.mp4 1 2 3 4 5 6 7 8 10 13 14 17 18). With no steps given, run_video.sh runs the full chain and run_image.sh runs the image-only subset. Steps 15/16/19 are optional and off by default. Steps 15 and 19 additionally need components this repository does not ship and you fetch yourself, one of them under academic-use-only terms — see the optional-components table in THIRD_PARTY.md before enabling them.

Default reconstruction backbone is uplegacy = legacy/Hi3DGen generation plus the upgraded pose chain (romav2 matching + zero-iteration scale recovery + all models resident in-process). It needs RoMaV2 and the Hi3DGen / SpaTrackerV2 / FoundationPose weights (all covered by step 5 above) and does not need TRELLIS2/CuMesh — those are only for --recon-backbone trellis2.

Runtime is ~19–22 min per video end-to-end, almost entirely compute (step 6 ~6 min + deform ~9–11 min + rendering); model loading is a one-off ~1–3 min service startup.

run_video_batch.sh and run_batch.sh start and stop opg-server + sam3-server automatically, each with its own queue — nothing to configure. Heavy models (SAM3, opg core, RoMaV2, FoundationPose) are then loaded once for the whole batch instead of once per video; Motion324 + Hunyuan (step 4) still load per video. Resident serving pays off across many videos — for a single video it is a wash, so single runs (run_image.sh / run_video.sh) stay in-process by default.

To manage the services manually across several batches:

bash scripts/servers/opg_server.sh  ensure --gpu 0   # ensure = start only if not already running
bash scripts/servers/opg_server.sh  status
bash scripts/servers/opg_server.sh  stop
bash scripts/servers/sam3_server.sh ensure --gpu 0

To make a single run reuse manually started services, export the queues explicitly. The engine treats "queue variable non-empty" as "use the server" and does not health-check it, so exporting a queue with no server running will hang until timeout:

export OPG_SERVER_QUEUE=/tmp/opg_server_main_q SAM3_SERVER_QUEUE=/tmp/sam3_server_q
bash scripts/run_video.sh --name eagle --video /path/eagle.mp4

All have sensible defaults in configs/paths.sh; override in .env.

Environment build (step 1) — the most likely thing to need per-platform tuning. The opg env compiles extensions on the machine (flash-attn / pytorch3d / FoundationPose). create_envs.sh handles each env independently and can be re-run per env (bash environment/create_envs.sh opg); the version table and the equivalent manual commands are in environment/README.md.

• nvcc not found → install CUDA Toolkit 12.1+ and put it on PATH.
• arch/ABI mismatch → confirm torch is the cu121 wheel and nvcc -V is ≥ 12.1.
• flash-attn compiles too slowly or fails → use an official prebuilt wheel matching cu121 + torch 2.4.

No DashScope account (step 2). Set QWEN_BACKEND="local" in .env to run step 1 on a local Qwen3-VL. The image path pulls Qwen/Qwen3-VL-8B-Instruct from HF automatically; you can also predownload with masking/download_qwen3_vl_8b_hf_mirror.sh and point at it with --qwen-model /path.

Placeholder expansion is one-shot (step 4). setup.sh runs configure.sh, which expands @ENGINE_DIR@, @OPG_PY@, @BLENDER_BIN@ and friends into the paths from your .env. Once expanded the tokens are consumed, so re-running it after editing .env will not update them. To reconfigure, restore the tree first:

git checkout -- engine/ third_party/ masking/ && bash configure.sh

Weights (step 5).

• facebook/sam3 is a gated HF repo: huggingface-cli login and accept its terms on the model page first.
• RoMaV2 weights need no download — torch.hub fetches them on first run.
• TRELLIS weights are only needed for --recon-backbone trellis2: OVOW_DL_TRELLIS2=1 bash download_ckpts.sh (+~20GB).
• The script creates two symlinks the engine expects: engine/checkpoints/OnePoseViaGen → ckpts/OnePoseViaGen and third_party/sam3/checkpoints/sam3/sam3.pt → ckpts/sam3/sam3.pt.

Runtime errors.

• server not ready → the service is still loading (opg takes ~1–3 min); check the service log in the queue directory.
• motion manifest missing (step 8) → step 5 did not run; make sure your step list includes 5.
• API key empty → check that APIKEY_FILE exists and contains sk-..., or switch to offline labeling with QWEN_BACKEND=local.
• Services will not stop → bash scripts/servers/opg_server.sh stop, or pkill -f opg_server.py.

If you find OVOW useful for your research, please cite:

@inproceedings{chen2026ovow,
  title     = {One Video, One World: Turning Monocular Video into Physical 4D Scenes},
  author    = {Chen, Junhao and Zhang, Boran and Chen, Mingjin and Zhang, Henghaofan and
               Zhang, Saining and Zhu, Congcong and Zhao, Hao and Huang, Ruqi and
               Li, Zhihao and Wang, Yufei},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

OVOW is an orchestration framework built on top of many excellent publicly released research works. We sincerely thank the authors of:

Core engine

• OnePoseviaGen (One View, Many Worlds) — the reconstruction engine of this project; OVOW's name is also a tribute to it.

Works integrated by the engine / depended on by this project

• FoundationPose — 6-DoF object pose estimation & tracking (NVIDIA)
• SpaTrackerV2 / SpatialTrackerV2 — 3D point tracking
• Hunyuan3D-2 — Hi3DGen 3D generation (Tencent Hunyuan)
• TRELLIS — 3D generative representation (Microsoft)
• Depth-Anything-3 (DA3) — monocular depth
• SAM 2 / SAM 3 — segmentation (Meta)
• Motion324 — deformable mesh animation
• RoMaV2 — dense feature matching (scale recovery / PnP correction)
• Amodal3R — occlusion-aware 3D reconstruction
• BundleSDF — pose refinement (NVIDIA)
• UniDepth — metric depth estimation (ETH)
• Qwen3-VL — multimodal labeling (Alibaba)

Per-component licenses and provenance: THIRD_PARTY.md.

OVOW-authored code is released under the MIT license (see LICENSE). However, third-party components bundled in this release carry non-commercial licenses, so the assembled repository uses mixed licenses and is for non-commercial research / evaluation use only — see NOTICE and THIRD_PARTY.md.