PQSG: Physics Question Scene Graph

Reference implementation for Physics Question Scene Graph (PQSG), a hierarchical question-based evaluation pipeline for measuring physical plausibility in text-to-video generation.

PQSG turns a text prompt into a directed acyclic graph of yes/no verification questions across three categories (Object → Action → Physics), and scores a generated video by asking a VLM to answer each question.

pip install -r requirements.txt
python scripts/reproduce.py

Reads cached PSGs and QA outputs from data/, recomputes Table 1 (overall correlation against human Likert) and Table 4 (model comparison) using pqsg.score.tree_score, and writes results to results/. No API keys needed.

To reproduce all paper tables/figures from cached data (no API calls):

python scripts/reproduce_all_tables.py   # Tables 1, 3, 4, 6, 8, 10, 11
python scripts/reproduce_videophy2.py    # Table 7 (VideoPhy-2 generalization)
python scripts/plot_refinement.py        # Figure 4 (iterative refinement)

export GOOGLE_API_KEY=...        # Gemini key for QG and QA
# (optional, only if --qa gpt5)
export OPENAI_API_KEY=...

python scripts/run.py \
  --input  examples/example_input.json \
  --output my_scores.json

Each prompt → generated PSG → VLM yes/no answers → tree-based PQSG score.

JSON list. Required fields per entry: prompt, video_path. Optional: id, model.

[
  {
    "id": 1,
    "model": "sora2",
    "prompt": "A blue grabber tool holds a tennis ball above a pile of green kinetic sand on a wooden table. The grabber then releases the ball.",
    "video_path": "/path/to/video1.mp4"
  },
  {
    "id": 2,
    "model": "veo3",
    "prompt": "A glass of water sits on a table. A drop of red food coloring falls into the water and slowly diffuses.",
    "video_path": "/path/to/video2.mp4"
  }
]

See examples/example_input.json.

Each input entry is augmented with psg, answers, and score:

{
  "id": 1,
  "prompt": "...",
  "video_path": "...",
  "psg": {
    "nodes": {
      "object_existence":     {"O1": "Is there a blue grabber tool?", ...},
      "action_verification":  {"A1": "Does the grabber release the ball?", ...},
      "physics":              {"P1": "Does the ball accelerate downwards?", ...}
    },
    "edges": [{"from": ["O1", "O2"], "to": "A1"}, ...]
  },
  "answers": {
    "object_existence": {
      "O1": {
        "question": "...",
        "initial_response": "Yes, a blue grabber tool is visible at...",
        "final_answer": "yes",
        "is_correct": true,
        "score": 1
      },
      ...
    },
    "action_verification": {...},
    "physics": {...}
  },
  "score": 0.85
}

Resume support: re-running scripts/run.py with the same --output skips entries that already have an answers field.

scripts/run.py
  --input,  -i  PATH   JSON list of {prompt, video_path} entries (required)
  --output, -o  PATH   destination for scored results (required)
  --qg          MODEL  question-generation model (default: gemini-2.5-flash)
                       supports: gemini-2.5-flash | gpt-5
  --qa          BACKEND  question-answering backend (default: gemini)
                         gemini  -> Gemini-2.5-Flash with native video input
                         gpt5    -> GPT-5 with extracted frames at 2 fps
  --qa-model    MODEL  override the QA model name

from pqsg import generate_psg, answer_psg_gemini, tree_score

psg = generate_psg("A ball drops onto sand")
result = answer_psg_gemini("video.mp4", psg)

print(result["score"])                                  # simple yes-ratio
print(tree_score(result["answers"], psg["edges"]))      # tree-based score

pqsg.score.category_scores(answers, edges) returns per-category scores restricted to questions whose parent chain is also correct.

.
├── README.md
├── requirements.txt
├── pqsg/
│   ├── __init__.py
│   ├── qg.py          # Question Generation
│   ├── qa.py          # Question Answering (two-step batched)
│   └── score.py       # Tree scoring with O→A→P error propagation
├── prompts/
│   └── qg_prompt.txt  # 7-example QG prompt used in the paper
├── data/              # cached PSGs, QA answers, human Likert scores
├── scripts/
│   ├── reproduce.py            # Tables 1 & 4 (quick)
│   ├── reproduce_all_tables.py # Tables 1, 3, 4, 6, 8, 10, 11
│   ├── reproduce_videophy2.py  # Table 7 (VideoPhy-2 generalization)
│   ├── plot_refinement.py      # Figure 4 (iterative refinement)
│   ├── build_finephyeval_manifest.py  # build the FinePhyEval release manifest
│   └── run.py                  # 1-command evaluation on user videos
├── examples/
│   └── example_input.json
├── data/
│   ├── finephyeval.json/.csv   # FinePhyEval dataset manifest (annotations + URLs + PSGs)
│   ├── human_likert_scores.csv # per-video human Likert scores
│   ├── human_qa_annotated_30.json  # human QA ground truth (Table 6/8)
│   ├── cached_psgs*.json, cached_qa_*.json  # cached PSGs / QA per backbone
│   └── videophy2/              # VideoPhy-2 generalization data (Table 7)
└── results/                    # populated by the reproduce_* scripts

Question Generation (QG). A VLM is prompted with seven in-context examples of (prompt, PSG) pairs (prompts/qg_prompt.txt) and produces a JSON PSG for a new prompt. Nodes are organized into Object / Action / Physics categories with explicit O→A and A→P dependency edges.

Question Answering (QA). Two-step batched QA: (1) one open-ended pass over all questions to get reasoning, (2) one forced yes/no pass over all questions to get categorical answers. Two API calls per video instead of 2×N.

Scoring. Tree-based: starting from any node answered "no", mark all descendant nodes "no" via forward edges. PQSG score = (yes after propagation) / (total nodes).

A note on edges: the paper's QG prompt nominally restricts edges to O→A and A→P. In practice the QG VLM also produces A→A (temporal action chains), O→P (skip-level), and P→P (physics chains), accounting for ~9% of all edges in our cached PSGs. These are valid forward dependencies and dropping them silently weakens the propagation signal. pqsg.score._is_valid_edge accepts all forward edges and only rejects true backwards ones (P→A, A→O, etc.). Using the strict-paper filter instead would lower Pearson by ~0.025; see scripts/ablate.py for the full grid.

See the paper for full details.

scripts/reproduce.py regenerates these from the cached data:

Table 1: PQSG vs human overall Likert (mean of object/action/physics, N=195)

QA Method               Pearson r  Kendall t  Spearman r
----------------------------------------------------------
Gemini-2.5-Flash            0.417      0.259       0.354
Gemini-2.5-Pro              0.467      0.306       0.406
GPT-5                       0.430      0.263       0.366
GPT-5.4                     0.429      0.274       0.369
GPT-5.5                     0.478      0.336       0.456    <- strongest

Both Gemini-2.5-Pro and GPT-5.5 strictly improve every metric over the published paper PQSG numbers (paper Gemini: 0.395 / 0.291 / 0.380 ; paper GPT-5: 0.435 / 0.265 / 0.375). GPT-5.5 is the recommended backend; Pro is the strongest open-Google option. See scripts/ablate.py for the full grid across edge filters and human targets.

The numbers above are produced by pure JSON->float computation on the cached data, with no API calls. They are bit-for-bit deterministic across runs:

pip install -r requirements.txt   # pinned versions
python scripts/reproduce.py --verify

--verify recomputes every Table 1 cell from data/*.json and asserts each value matches a frozen expected value to within 0.001. CI/collaborators can use this as a regression gate.

For the live pipeline (scripts/run.py) on user videos, both QG and QA call the VLMs at temperature=0 to minimize stochasticity, but the providers do not guarantee bit-exactness so re-runs may produce slightly different yes/no answers (typically <2% per video). To get reproducibility-by-cache, save the output JSON from run.py and rerun the scoring step (pqsg.score.tree_score) on it whenever you need the numbers again.

FinePhyEval contains 195 prompt-video pairs (65 prompts sourced from Physics-IQ, generated by Sora 2, Veo 3, and Wan 2.1) with fine-grained human Likert annotations across object, action, physics, and overall categories, plus the PQSG graph for each prompt. We additionally release the 65 Cosmos-Predict2.5-14B videos on the same prompts (model comparison, Table 4); these are evaluated by PQSG but, unlike the three core models, have no human Likert annotations (human is null in the manifest).

• Manifest: data/finephyeval.json (260 records, 65 each for Sora 2, Veo 3, Wan 2.1, and Cosmos, with prompt, public video URL, the four human scores where available, and the PSG) and data/finephyeval.csv (flat annotations table). Build/refresh with python scripts/build_finephyeval_manifest.py.
• Videos: all 260 are hosted publicly on S3 at https://atin-videos.s3.amazonaws.com/... (referenced by video_url in the manifest). Verify reachability with python scripts/build_finephyeval_manifest.py --check-live.
• Annotations: data/human_likert_scores.csv (per-category averages and number of annotators per video).

@inproceedings{pothiraj2026pqsg,
  title     = {Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation},
  author    = {Pothiraj, Atin and Cho, Jaemin and Zhang, Yue and Stengel-Eskin, Elias and Bansal, Mohit},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}