This repository extends CLIPSitu to the VidSitu dataset, enabling video-based situation recognition. CLIPSitu-Vid builds upon CLIPSitu’s CLIP-based semantic role labeling, integrates the VidSitu dataset, and evaluates Vision-Language Models (VLMs) like VILA via zero-shot prompting.
This work is based on our IJCV journal paper:
📄 Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos (2024).
CLIPSitu-Vid is designed to perform multi-frame situation recognition, where it predicts verbs, roles, and objects at multiple timestamps within a video.
- Video Situation Recognition: Predicts structured event representations (verbs, roles, objects) for each 2-second segment in a video.
- Transformer with Temporal Encoding: Extends CLIPSitu’s image-based CLIP embeddings with video-aware multi-frame reasoning.
- VLM Evaluation for Benchmarking: Compares zero-shot prompting from VILA against our structured approach.
- Structured Outputs: Generates JSON-based outputs, compatible with downstream reasoning systems.
- A large-scale video dataset (29,000 movie clips) annotated at 2-second intervals with verbs, roles, and entities.
- Each video contains five events, with semantic role labels for each verb.
- For more details, visit the VidSitu repo.
- We build upon GVSR, a graph-based model for video-based semantic role labeling.
- GVSR enhances verb and role prediction with a graph-based temporal structure.
- More details available in the GVSR repo.
Large datasets and models are stored on Google Drive to keep the repository lightweight.
Download the required files from the link below:
📂 Google Drive: CLIPSitu-Vid Datasets & Models
CLIPSitu-Vid utilizes CLIP embeddings to extract rich visual-linguistic representations from individual video frames. These embeddings are fed into a temporal attention-based transformer, which:
- Aggregates multi-frame context using self-attention.
- Predicts structured outputs (verb, roles, objects) at each timestamp.
- Uses learned constraints to improve noun-role assignments.
To compare CLIPSitu-Vid with a Vision-Language Model (VLM), we use VILA for zero-shot prompting in Jupyter notebooks.
📄 VILA is only used as a benchmark and is not integrated into the main model.
This research/project is supported by the National Research Foundation, Singapore, under its NRF Fellowship (Award# NRF-NRFF14-2022- 0001). This research is also supported by funding allocation to B.F. by the Agency for Science, Technology and Research (A*STAR) under its SERC Central Research Fund (CRF), as well as its Centre for Frontier AI Research (CFAR).
This project builds upon:
- CLIPSitu (leveraging CLIP for image-based situation recognition).
- VidSitu (large-scale video semantic role labeling dataset).
- GVSR (Graph-based Video Situation Recognition).
We thank the authors of these projects for their foundational contributions.
If you use this work, please cite:
@article{verma2024effectively,
title={Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos},
author={Verma, Dhruv and Roy, Debaditya and Fernando, Basura},
journal={International Journal of Computer Vision},
url={https://arxiv.org/pdf/2407.20642},
year={2025}
}
@inproceedings{roy2024clipsitu,
title={Clipsitu: Effectively leveraging clip for conditional predictions in situation recognition},
author={Roy, Debaditya and Verma, Dhruv and Fernando, Basura},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={444--453},
year={2024}
}
@article{VidSitu2021,
title={VidSitu: A Large-Scale Video Dataset for Situation Recognition},
author={Sadhu, A. and others},
journal={CVPR},
year={2021}
}
@article{GVSR2021,
title={GVSR: A Graph-Based Model for Video Situation Recognition},
author={Zeeshan, K. and others},
journal={arXiv},
year={2021}
}- Fine-tuning CLIP-based transformers for multi-frame attention.
- Logic-based constraints to improve role assignments.
- Multi-modal learning: Incorporating audio cues for enhanced understanding.
