Under review Robot world models Event-level diagnosis Training-free

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

A single score cannot say what went wrong. For every visible failure in a generated robot video, RoboGaze reports what failed, when, why, and how severely.

Minh-Loi Nguyen1, Nghiem T. Diep2, Hung K. Nguyen2, Minh Le2, Doanh L. Thien1,2, Tuan A. Tran2, Hoang H. Tran1, Dung D. Le1, Vu N. Duong1, Khoa D. Doan2, Ngan Le7, Daniel Sonntag3, An T. Le1,2,6, Vien Anh Ngo1,2, Duy M. H. Nguyen†,3,4,5, Tran V. Nhiem†,2
1Center for AI Research, VinUniversity, 2VinRobotics, 3DFKI, 4University of Stuttgart,
5Max Planck Research School for Intelligent Systems, 6Technische Universität Darmstadt, 7University of Arkansas, USA

† Project Leads

Paper Code Benchmark soon
+11.3
description F1 over the strongest baseline, mean of 8 VLM backbones
17%→85%
correct empty reports on failure-free clips, chain-of-thought vs. RoboGaze
83%
of the gap between chain-of-thought and human agreement closed (Gemini 3.1 Pro)
914
generated clips in RoboGazeBench, with 1,847 time-stamped failures
Left: a generated clip in which a robot hand passes through a cup between 2.4 and 3.1 seconds. Middle: a score-based evaluator gives 4 out of 5 and misses the failure, while the RoboGaze diagnosis states what (object penetration), when (2.4 to 3.1 s), why and severity 4 of 5. Right: bar charts of event description F1 and clean-clip accuracy for CoT, GLiDE+ and RoboGaze, with dashed lines for human agreement.
A score misses it. RoboGaze explains it. A judge rates this clip 4/5 and misses the hand passing through the cup. Results on Gemini 3.1 Pro; RG is RoboGaze.
RoboGaze in two minutes. The problem, a diagnosis as the clip plays, the method, the benchmark and the results.
01Demo

Every failure, at the moment it happens.

Play a generated clip. RoboGaze’s findings appear as the video reaches each failure, with its type, time span, dimension and severity. Open a finding to see the frames behind it.

paused
0.0 s
02Method

Ground the task, route to specialists, verify.

One frozen VLM plays every role, with no task-specific training. It grounds the instruction and the scene, sends each subtask only to the failure specialists that matter, and lets a critic accept, merge or reject every candidate before it enters the report.

RoboGaze framework in three phases: input and context grounding with task memory, scene memory, an observer over fixed-length windows and subtask segments; candidate discovery where a router sends suspicious spans to six dimension-specific specialists that produce failure hypotheses; verification and reporting where a critic verifier accepts, merges or rejects hypotheses and synthesizes the final report with dimension, description, severity and time.
Three phases. Ground the task and scene, send each subtask to at most three of six specialists, then verify every hypothesis before it enters the report.
The pipeline, step by step on one benchmark clip (40 s loop). The left panel shows what the module is looking at; the right panel shows which module is running and what it returns. Intermediate outputs are abridged for display.
03Benchmark

RoboGazeBench: 914 generated videos with time-stamped failures.

Generated manipulation videos from five generators and three robot embodiments. Annotators label each visible failure with its type, time span, severity and a description. 210 clips contain no failure, so an evaluator must also know when to stay silent.

914generated clips, 210 failure-free
1,847annotated failure events
5generators, 2 held out from development
3embodiments, 194 instructions
4.9–148.9 sclip length, median 10.4 s
κ = .74annotator agreement on whether a clip fails

Three embodiments

Humanoid, egocentric viewCosmos-Predict2.5
Bimanual, top-down viewDreamDojo, shown at 2.5× speed
Single arm, front viewDreamDojo

What the clips cover

Clip counts by task family, generator, length and embodiment. Each clip is a generated example from its task family.

T1
Free-space transport567 clips, 62%pick, place, move, hold
T21.5× speed
Constrained contact81 clips, 9%pour or scoop, insert or assemble, tool use, reorient
T31.5× speed
Articulated & activation155 clips, 17%buttons and switches, open or close
T43× speed
Coordinated & deformable111 clips, 12%fold, hand over, clean or wipe

Generators

* held out from all development

Clip length

median 10.4 s, longest 148.9 s

Embodiment

Six dimensions, thirty failure types

Each dimension asks one question. In these benchmark clips the frame lights up while the annotated failure is happening.

Task semantics

0.7× speed
Task Progress412 events

Did the task reach the required state?

Types: Task incompletion, failed grasp, failed placement, premature termination, ambiguous task success.

0.75× speed
Instruction Consistency198 events

Did it follow the instruction and its constraints?

Types: Wrong effector, wrong object, wrong target location, wrong action order, ignored constraint.

Entity consistency

1.5× speed
Object–Scene Consistency341 events

Do the objects stay coherent?

Types: Object hallucination, object disappearance, object identity swap, object distortion, object appearance drift.

0.9× speed
Robot-Body Consistency286 events

Does the robot keep its visible structure?

Types: Hallucinated robot part, missing robot part, duplicated arm or gripper, robot body deformation, left/right identity confusion.

Physics & observability

0.6× speed
Physical Plausibility379 events

Is the visible interaction physically possible?

Types: Object teleportation, object floating, object penetration, grasp without visible support, other implausible dynamics.

142 s clip at 10× speed
Visual Quality231 events

Are the frames clear enough to judge?

Types: Frame corruption, camera instability, blur, occlusion, low visibility.

04Results

Better diagnoses on every backbone.

All methods use the same frozen VLM and the same video input. GLiDE+ is the strongest control: a published event detector given RoboGaze's schema and severity rubric.

+11.3F1 points

RoboGaze describes failures better than every baseline, on all eight VLMs.

Event description F1 on all 914 clips. The gain over GLiDE+ stays between 10.1 and 12.3 points, from an 8B open model to Gemini 3.1 Pro.

CoT promptingGLiDE+RoboGazehuman agreement
85%

It stays silent when nothing failed.

Correct empty reports on the 210 failure-free clips, mean of 8 backbones.

+16.6on clips over 80 s

The margin grows on harder clips.

F1 gain over GLiDE+ by clip length. On the held-out generators it is +12.4 (Wan 2.2) and +13.0 (Veo).

71.6vs. 61.4 F1

Blind human raters confirm it.

Human re-scoring of the gold subset, and precision when the reports are used to select generated demonstrations.

Event F1 re-scored by humans

Precision of selected demonstrations

05Scope
Visible failures only

An empty report means no failure is visible in the sampled frames. It does not show that the behavior is executable on a robot.

One camera view

Benchmark clips are single-view; consistency across cameras is not evaluated.

Several VLM calls per clip

The paper compares RoboGaze with controls given an equal or larger inference budget.

06Use & extend

Three ways to use RoboGaze.

Diagnose your own generated videos, benchmark another evaluator on RoboGazeBench, or extend the framework with a new backbone, failure type or data. None of them requires training.

Using and extending RoboGaze. A, diagnose your videos: inputs (generated video, first frame, task instruction), RoboGaze on a frozen VLM (ground, route, verify), and a time-stamped report. B, benchmark your evaluator: load RoboGazeBench, run your evaluator and export events, match them to human references in time and description, and compare with CoT, GLiDE+ and RoboGaze. C, extend the framework with a new VLM backbone, a new failure type or new data.
Using and extending RoboGaze. Lane A is the main use and needs no reference annotations. Lanes B and C use the benchmark and code, which will be released with the paper.
A

Diagnose your videos

1Inputs

ImageImage

“Press the pink, then the blue, then the green button.”

A generated video, its first frame and the task instruction.

2Run RoboGaze on a frozen VLM

Ground→Route→Verify

One frozen VLM plays every role. No fine-tuning.

3Read the report

Object disappearance12.0–19.0 sObject–Scene ConsistencySeverity 4 / 5

One report per clip. A clean clip gets an empty list.

B

Benchmark your evaluator

1Load the data

914 clips, 1,847 failures, 210 clean clips.

2Run yours

Export one event list per clip: dimension, type, start and end time, severity, description.

3Match events

Events are matched to the human reference by time overlap and description.

4Compare

Description F1, temporal IoU and clean-clip accuracy, next to CoT (23.5), GLiDE+ (46.0) and RoboGaze (57.3).

C

Extend the framework

New backbone

Wrap any VLM behind one call: frames and a prompt in, text out.

New failure type

Add its definition and a specialist prompt. Scoring needs new annotations.

New data

Add clips from a new generator or robot with reference annotations, then score them as in B.

Citation

BibTeX

@misc{nguyen2026robogaze,
  title   = {RoboGaze: Evaluating Robot World Models via
             Structured Vision-Language Analysis},
  author  = {Nguyen, Minh-Loi and Diep, Nghiem Tuong and Nguyen, Hung Khang and
             Le, Minh and Le Thien, Doanh and Tran, Hoang H. and Le, Dung D. and
             Duong, Vu N. and Sonntag, Daniel and Le, An Thai and Nguyen, Duy Minh Ho and
             Ngo, Vien Anh and Nhiem, Tran Van},
  year    = {2026},
  eprint  = {2606.28385},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  doi     = {10.48550/arXiv.2606.28385},
  url     = {https://arxiv.org/abs/2606.28385}
}