HLV-1K is a comprehensive benchmark designed to evaluate the capabilities of multimodal large language models (MLLMs) in understanding hour-long videos with time-specific queries. Unlike existing video understanding benchmarks that focus on short clips, HLV-1K addresses the critical challenge of long-term video comprehension by providing:

• 🕐 Hour-long Videos: 1,009 videos with an average duration of 1 hour
• 📊 Diverse Reasoning Tasks: 14,847 QA and MCQA pairs across multiple reasoning levels
• ⏰ Time-specific Queries: Questions that require understanding of specific temporal segments
• 🎯 Multi-level Evaluation: Frame-level, within-event, cross-event, and long-term reasoning

As video content becomes increasingly prevalent and lengthy, HLV-1K provides a robust evaluation framework for assessing models' ability to comprehend and reason about extended video sequences with precise temporal understanding.

Accuracy scores on HLV-1K are presented on frame-level, within-event-level, cross-event-level and long-term-level.

• 📹 Video Scale: 1,009 hour-long videos (average duration: ~1 hour)
• ❓ Question Diversity: 14,847 QA and MCQA pairs with time-specific queries
• 🔍 Multi-level Reasoning: Four distinct reasoning levels for comprehensive evaluation
• ⏱️ Temporal Precision: Questions anchored to specific time segments within videos

Benchmark construction and examples.

HLV-1K: (a) Video category distribution, (b) Video duration distribution, and (c) Duration distribution of time-specific query.

HLV-1K: Distribution of benchmark annotations.

HLV-1K employs a sophisticated annotation pipeline using GPT-4o for high-quality question generation:

1. Frame Description Extraction: Detailed descriptions of video frames at specific timestamps
2. Event Summarization: Coherent event descriptions spanning ~60 seconds with precise temporal boundaries
3. Question Generation: Time-specific questions across four reasoning levels
4. Quality Assurance: Multi-round validation to ensure question accuracy and temporal precision

• Accuracy: Overall correctness across all question types
• Level-wise Performance: Accuracy breakdown by reasoning level
• Task-specific Metrics: Performance on different cognitive tasks
• Temporal Understanding: Accuracy on time-specific queries

Evaluation results of four representative MLLMs.

Comprehensive-Long-Video-Understanding-Survey

The HLV-1K dataset is available for research purposes. Please follow these steps:

1. Clone the repository:

   git clone https://github.com/Vincent-ZHQ/HLV_1K.git
   cd HLV_1K

2. Dataset structure:

   HLV_1K/
   ├── dataset/           # 1,009 JSON files with QA pairs
   ├── static/           # Web interface assets
   ├── gpt_evaluation.py # Evaluation script
   └── index.html        # Web interface
   

1. Load dataset:

   import json
   
   # Load a single video's QA pairs
   with open('dataset/video_id.json', 'r') as f:
       qa_pairs = json.load(f)
   
   for qa in qa_pairs:
       print(f"Question: {qa['question']}")
       print(f"Answer: {qa['answer']}")
       print(f"Level: {qa['qa_level']}")
       print(f"Task: {qa['qa_task']}")

2. Evaluation:

   python gpt_evaluation.py --model_name your_model --results_file your_results.json

Each JSON file contains QA pairs with the following structure:

{
  "qa_idx": 1,
  "qa_type": "mcqa",
  "qa_level": "within_event",
  "qa_task": "speed",
  "question": "Are the individuals working at a fast pace between 1290.0 and 1350.0 seconds?",
  "answer": "No",
  "options": ["A. Yes", "B. No"]  // For MCQA only
}

We welcome contributions to improve HLV-1K! Please feel free to:

• Report issues or bugs
• Suggest new features or improvements
• Submit pull requests

This project is licensed under the MIT License - see the LICENSE file for details.

If you find our work helpful, please consider citing:

@article{zou2025hlv,
  title={Hlv-1k: A large-scale hour-long video benchmark for time-specific long video understanding},
  author={Zou, Heqing and Luo, Tianze and Xie, Guiyang and Zhang, Victor Xiao Jie and Lv, Fengmao and Wang, Guangcong and Chen, Junyang and Wang, Zhuochen and Zhang, Hansheng and Zhang, Huaijian},
  journal={arXiv preprint arXiv:2501.01645},
  year={2025}
}

We thank all contributors and the research community for their valuable feedback and support in developing HLV-1K.