Official implementation of "Advancing Reference-free Evaluation of Video Captions with Factual Analysis"

• Key Features
• Installation
• Inference
• Dataset
• Synthetic Data Generation
• Training
• Results
• Citation
• Contact

• Reference-free Evaluation: Evaluates video captions without requiring ground-truth references
• Factual Grounding: Focuses on detecting factual errors in objects and actions
• Interpretable Outputs: Generates quality scores (1-5) with natural language explanations
• Lightweight & Open-source: Built on Qwen2.5-VL (3B/7B), fully reproducible
• Cross-domain Generalization: Works on both video and image caption evaluation

# Clone the repository
git clone https://github.com/dipta007/VC-Inspector.git
cd VC-Inspector

# Install all dependencies
pip install -e .

# Then install PyTorch for your CUDA version (see https://pytorch.org/get-started/locally/)
# Example for CUDA 12.4:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

The training dataset is available on HuggingFace:

from datasets import load_dataset

dataset = load_dataset("dipta007/ActivityNet-FG-It", "balanced")

Each example contains:

• messages: Chat-formatted data with 32 <image> tags, a caption, an evaluation prompt, and a score + explanation
• images: List of 32 video frame paths

The dataset is generated by systematically augmenting ground-truth ActivityNet captions with controlled factual errors in objects and actions. Each sample pairs a video (as 32 frames) with a caption and a quality score (1-5) along with a natural language explanation.

Our data generation pipeline creates training samples by systematically augmenting ground-truth captions with controlled factual errors.

1. Extract objects and actions from ground-truth captions using an LLM
2. Generate alternative objects/actions with different meanings
3. Substitute original elements with alternatives
4. Compute quality scores based on semantic similarity
5. Generate explanations identifying the factual errors

Step 1: Start the vLLM servers

# Start vLLM server for text generation
vllm serve Qwen/Qwen3-32B --port 8000 --trust-remote-code

# Start vLLM server for embeddings (quality scoring)
bash src/gen_data/embedding.sh

Step 2: Run the generation script

python -m src.gen_data.gen \
    --gt_file data/actnet/train.caption_coco_format.json \
    --dataset anet \
    --n_sample 10 \
    --out_dir data/synthetic_data \
    --gen_model_id Qwen/Qwen3-32B \
    --num_of_workers 64

Arguments:

We fine-tune vision-language models using LoRA with the ms-swift framework.

Training uses the dipta007/ActivityNet-FG-It dataset (balanced subset) from HuggingFace Hub directly.

bash src/training/train_lora.sh \
    "Qwen/Qwen2.5-VL-3B-Instruct"

Supported Base Models:

• Qwen/Qwen2.5-VL-3B-Instruct
• Qwen/Qwen2.5-VL-7B-Instruct

python -m src.inference \
    --model dipta007/VCInspector-7B \
    --video path/to/video.mp4 \
    --caption "A man is playing guitar in a field"

Example Output:

4
The caption does not accurately capture the video content. For example, the objects (guitar) are incorrect.

If you find this work useful, please cite our paper:

@inproceedings{dipta2026vcinspector,
  title={VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis},
  author={Shubhashis Roy Dipta and Tz-Ying Wu and Subarna Tripathi},
  booktitle={Proceedings of the Association for Computational Linguistics: ACL 2026},
  year={2026},
  eprint={2509.16538},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2509.16538},
}

For questions or issues, please open a GitHub issue or contact:

• Shubhashis Roy Dipta: sroydip1@umbc.edu