A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs

Yifei Li^1,2,†  ·  Pengyiang Liu^3,†  ·  Yuhang Zang^2,*  ·  Zhongyue Shi³  ·  Qi Fu³  ·  Hongye Hao³  ·  Jiwen Lu¹

¹Tsinghua University    ²Shanghai AI Laboratory    ³Beihang University
^†Equal Contribution    ^*Project Leader

Overview of OVO-S-Bench. The benchmark evaluates streaming spatial understanding across four levels, from instantaneous egocentric perception and spatiotemporal context tracking to generative spatial reasoning and global topological mapping. The right panel summarizes representative model behavior across task families.

Multimodal agents in robotics, AR, and autonomous driving must reason about places and layouts from continuous egocentric streams, often using evidence outside the current view. Existing benchmarks either evaluate offline over full videos or target events rather than spatial structure. We introduce OVO-S-Bench, a fully human-annotated benchmark for streaming spatial intelligence, comprising 1,680 questions over 348 source videos. Annotation involves 12 trained annotators (each also serving as a blind cross-reviewer) across roughly 804 person-hours of multi-round quality assurance. Each question carries a query timestamp and an evidence interval, and at evaluation, the model sees only the prefix preceding the query. Questions span four levels of increasing abstraction: instantaneous egocentric perception, spatiotemporal context tracking, spatial simulation and reasoning, and allocentric mapping. Across 38 proprietary and open-source MLLMs, Gemini-3.1-Pro trails human experts by 27 points (59.2 vs. 86.6), with allocentric mapping as the dominant bottleneck. Notably, streaming and spatially fine-tuned MLLMs underperform their own backbones. We further find that chain-of-thought reasoning amplifies spatial errors when ungrounded in the stream. By exposing these limitations, OVO-S-Bench establishes a demanding testbed for next-generation streaming spatial MLLMs.

OVO-S-Bench organizes questions into four cumulative levels by the spatial state a model must access at query time. The progression goes from evidence directly available in the current view to allocentric map queries that require cross-viewpoint integration, reflecting a gradient of persistence and abstraction.

The released benchmark comprises 1,680 questions over 348 source videos from 9 datasets, organized into 30 canonical task types across four levels. Mean prefix at query time: 8.8 min. Evidence-span medians by level: L1 2.0 s · L2 36.8 s · L3 2.0 s · L4 278.7 s — reflecting the spatial persistence each level demands.

Representative OVO-S-Bench examples. Each card pairs a spatial question with visual evidence, illustrating the progression from current-view perception to allocentric mapping.

Taxonomy and benchmark statistics. Left: four-level spatial taxonomy. Right: task-family counts, source distribution, and evidence-interval lengths by level.

L4.3 trajectory-matching questions ship their option images embedded inline as base64 data URIs in the parquet's options column. No separate image-asset download is needed.

• Video sources. OVO-S-Bench draws from 9 publicly available or accessible sources covering five regimes: indoor walkthroughs (RoomTour3D), egocentric activities (Ego4D), outdoor/world scenes (Sekai, OmniWorld, YouTube walking tours), driving videos (CODa, Honda HDD), and spatially annotated 3D environments (ARKitScenes, VSI-Bench).
• Human annotators write every item. Annotators with 3D-vision backgrounds choose clips with stable motion, clear viewpoints, and enough spatial variation for the target level. For each item, they record the video, task label, question, options, answer, query timestamp, and evidence interval.
• Each item follows the streaming setting. The answer must be derivable from the video prefix before the query timestamp. Annotators mark the shortest interval that contains the needed evidence and write distractors that are plausible under the visual context but wrong under the annotated evidence.
• Quality control removes shortcuts. A text-only LLM probe flags items that leak the answer through wording, common sense, or option asymmetry. A second annotator then cross-reviews each item without seeing the original answer, checking that the answer and evidence interval are sufficient. Recurring problems are folded back into the annotation guideline.

Six observations about the current state of streaming spatial intelligence, from 38 evaluated systems on OVO-S-Bench:

Main results under the streaming protocol (multiple-choice accuracy). Numbers replicated from the paper's main results table. The public dataset release is linked above; a live submission portal will follow.

Interactive version with sorting + per-category drill-down: https://internlm.github.io/OVO-S-Bench/

git clone https://github.com/InternLM/OVO-S-Bench.git
cd OVO-S-Bench

# Default install (API providers only; ~1 min)
pip install -r requirements.txt

# Optional: open-source MLLMs via vLLM
pip install -r requirements-vllm.txt

The release pages are:

• Hugging Face: https://huggingface.co/datasets/JoeLeelyf/OVO-S-Bench
• ModelScope: https://modelscope.cn/datasets/JoeLeelyf/OVO-S-Bench/

The complete annotations parquet and source videos are currently available from ModelScope:

pip install -U modelscope
python - <<'PY'
from modelscope.hub.snapshot_download import snapshot_download

snapshot_download(
    repo_id='JoeLeelyf/OVO-S-Bench',
    repo_type='dataset',
    local_dir='./data',
)
PY

Layout you should see after download:

data/
├── ovo_s_bench.parquet     # questions, ~35 MB
└── videos/                       # source .mp4 files
    ├── Ego4D/ ...
    ├── RoomTour3D/ ...
    ├── annotated_videos/ ...
    └── ...

cp .env.example .env
$EDITOR .env       # fill in the provider(s) you'll use

API model (single-process, threaded):

bash scripts/eval_api.sh gpt-4o data/ovo_s_bench.parquet

Open-source MLLM via vLLM (multi-GPU auto-sharded; default 128 uniformly-sampled frames per prefix):

GPUS=8 bash scripts/eval_vllm.sh qwen3-vl-32b data/ovo_s_bench.parquet

Both scripts call inference.py then score.py. Inference auto-resumes on rerun — interrupted jobs pick up from the last checkpoint.

See docs/adding_models.md. The contract is BaseModel.inference(frames: List[PIL.Image], prompt: str) -> str; add a config entry to config.yaml::MODELS and register the wrapper in models/api_models.py or models/vllm_models.py.

Set *_BASE_URL in .env to point at your proxy or local vLLM server. API providers (openai, gemini-native, anthropic) honor the standard *_BASE_URL env var convention.

All systems are evaluated under a unified streaming protocol: each source video is truncated at the annotated query timestamp $t_q$, and the model receives 128 frames uniformly sampled from the resulting prefix together with the question and multiple-choice options. For streaming-architecture models that implement a native sequential ingestion path, we instead feed the video at each model's published streaming rate and query the resulting compressed state. No model sees frames after $t_q$. Answers are extracted by regular expression without further post-processing.

Per-query responses land under results/<model>/ovo_s_bench.json. score.py aggregates accuracy by main category and subcategory:

python score.py --result results/gpt-4o/ovo_s_bench.json --verbose

For sharded multi-GPU runs, merge_results.py consolidates *_rank{N}.json shards into a single result file (auto-invoked by launch.py).

OVO-S-Bench/
├── inference.py            # Main eval entry point
├── score.py                # Per-category accuracy aggregator
├── launch.py               # Multi-GPU dynamic-sharding launcher (auto-merges)
├── precache.py             # CPU-only frame pre-extraction
├── merge_results.py        # Shard merger
├── prompts.py              # Pluggable prompt templates (default/verbose/cot)
├── annotation_utils.py     # Parquet / JSON annotation loader
├── option_utils.py         # Image-option helpers (L4.3 PNG decoding)
├── config.yaml             # 84+ pre-registered model entries
├── models/
│   ├── base.py             # BaseModel ABC
│   ├── api_models.py       # OpenAI / Gemini / Claude
│   ├── vllm_models.py      # Qwen3-VL / Qwen3.5 / Gemma4 / GLM-4.6V
│   ├── internvl_models.py
│   ├── llava_onevision_vllm_models.py
│   ├── minicpmv_models.py
│   └── extras/             # 12 wrappers requiring upstream repos
│       ├── README.md       # BYO install instructions
│       └── *_models.py     # HERMES / FluxMem / StreamingTOM / InfiniPot-V / ...
├── utils/
│   ├── frame_utils.py      # decord + OpenCV fallback, .frame_cache logic
│   └── config_utils.py     # nested → flat config flatten
├── scripts/
│   ├── eval_api.sh
│   └── eval_vllm.sh
├── data/README.md          # HF dataset download instructions
└── docs/
    ├── adding_models.md
    ├── benchmarking_protocol.md
    └── frame_caching.md

@misc{li2026ovosbench,
  title         = {OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs},
  author        = {Li, Yifei and Liu, Pengyiang and Zang, Yuhang and Shi, Zhongyue and Fu, Qi and Hao, Hongye and Lu, Jiwen},
  year          = {2026},
  eprint        = {2606.03890},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.03890}
}

OVO-S-Bench draws videos from publicly released datasets — many thanks to: Ego4D, RoomTour3D, CODa, OmniWorld, VSI-Bench, Sekai, ARKitScenes, Honda HDD, and selected YouTube walking tours.

The evaluation framework integrates wrappers for several token-compression and streaming MLLM research repos: HERMES, FluxMem, StreamingTOM, InfiniPot-V, InfiniteVL, Flash-VStream, StreamForest, Spatial-MLLM, Cambrian-S, SenseNova-SI, Spatial-TTT.

Project page template adapted from OVO-Bench, which itself builds on the Nerfies template.

MIT — see LICENSE. Annotations released under CC-BY-4.0; source videos retain their original licenses.