Advances in Neural Information Processing Systems (NeurIPS), 2026
[📜 Paper] [🌐 Project Page] [🤗 Dataset] [🤗 Model]
Junho Kim1, Xu Cao1, Houze Yang1, Bikram Boote1, Ana Jojic1, Fiona Ryan2, Bolin Lai3, Sangmin Lee4, James M. Rehg1
1UIUC · 2Georgia Tech · 3Amazon AGI · 4Korea University
GRASP is a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events — 290K question–answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze–gesture reasoning, together with GRASP-Bench for evaluation. We also propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction.
- Paper release
- Project page
- Training code (SFT + SGR)
- GRASP dataset and GRASP-Bench
- Model weights (GRASP-Qwen3-VL-8B)
Requires Python 3.12 and CUDA 12.
curl -LsSf https://astral.sh/uv/install.sh | sh
uv syncSupported models: Qwen/Qwen3-VL-8B-Instruct, Qwen/Qwen3.5-9B.
Download interlive/GRASP and extract the video archives:
hf download interlive/GRASP --repo-type dataset --local-dir data/GRASP
mkdir -p data/videos data/grasp_bench
for t in data/GRASP/train/videos/*.tar; do tar -xf "$t" -C data/videos; done
tar -xf data/GRASP/test/grasp_bench.tar -C data/grasp_benchDATA_PATH=data/GRASP/train/data_sft.jsonl VIDEO_ROOT=data/videos \
bash scripts/train_sft.shDATA_PATH=data/GRASP/train/data_rl.jsonl VIDEO_ROOT=data/videos \
MODEL_PATH=checkpoints/grasp_sft \
bash scripts/train_sgr.shThe reward combines answer correctness, response format, and participant-level grounding (correct / has_format / has_grounding / grounding_quality = 1.0 / 0.1 / 0.05 / 0.2), computed against the ground-truth social events (gt_events) attached to each MCQ sample.
| Variable | Description |
|---|---|
MODEL_PATH |
HuggingFace model ID or local checkpoint |
MODEL_TYPE |
qwen3_vl, qwen3_5 |
DATA_PATH |
Training JSONL |
VIDEO_ROOT |
Directory the video tars were extracted into |
NUM_GPUS |
GPUs for DeepSpeed ZeRO-2 (default: 8) |
OUTPUT_DIR |
Checkpoint output directory |
BENCH_DIR=data/grasp_bench MODEL_PATH=interlive/GRASP-Qwen3-VL-8B \
bash scripts/eval_grasp_bench.shPer-category accuracy (Gaze / Gesture / Joint) is printed at the end and predictions are saved under eval/results/.
Code, annotations, and model weights are released under CC BY-NC 4.0 for non-commercial research use. Video clips remain subject to the licenses of their original source datasets.
@article{kim2026grasp,
title={GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions},
author={Kim, Junho and Cao, Xu and Yang, Houze and Boote, Bikram and Jojic, Ana and Ryan, Fiona and Lai, Bolin and Lee, Sangmin and Rehg, James M},
journal={arXiv preprint arXiv:2605.15764},
year={2026}
}