CoPE-VideoLM
Abstract
Replace dense per-frame embeddings with codec primitives.
Video Language Models (VideoLMs) enable AI systems to understand temporal dynamics in videos. To fit within the maximum context window constraint, current methods use keyframe sampling which often misses both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their tokens for each frame incurs substantial computational overhead. We address these limitations by leveraging video codec primitives (specifically motion vectors and residuals) which natively encode video redundancy and sparsity without requiring expensive full-image encoding for most frames.
To this end, we introduce lightweight transformer-based encoders that aggregate codec primitives and align their representations with image encoder embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach, CoPE-VideoLM, reduces the time-to-first-token by up to 86% and token usage by up to 93% compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we maintain or exceed performance on 14 diverse video understanding benchmarks spanning general question answering, temporal and motion reasoning, long-form understanding, and spatial scene understanding.
Interactive Demo
User chat experience comparing LLaVA-Video and CoPE-VideoLM at 1 FPS video input.
Token Efficiency vs. Video QA Accuracy
Performance of LLaVA-Video (7B) at different numbers of keyframes per GOP, as well as in the default setup of selecting 64 keyframes regardless of video length, compared to CoPE-VideoLM.
Benchmark Performance
Comprehensive evaluation covering general video question answering, temporal reasoning and motion understanding, long-form and instruction-following tasks, and spatial scene understanding.
Primary comparison with LLaVA-Video-7B alongside closely related open-source approaches, all evaluated using lmms-eval for consistency.
Latency and Scaling
TTFT and E2EL for generating 64 text tokens at several keyframe densities compared to the 64 keyframe baseline at 1 FPS.
Runtime Comparison
Token Budget vs. Video Length
Theoretical plot showing token efficiency across configurations — enabling scaling to longer videos without exceeding context limits.
Methodology
Given a video in its raw codec representation, our framework leverages the GOP structure for efficient, codec-aware tokenization.
I-frames are processed by a standard frozen vision encoder (φRGB) to produce dense RGB tokens. P-frames, however, bypass full RGB decoding. Their raw components, motion vectors and residuals, are instead fed into our lightweight Δ-Encoder (φΔ) to generate a small set of highly compact Δ-tokens. The final token stream, an interleaved sequence of I-frame tokens and Δ-tokens, is consumed by the LLM, enabling dense temporal coverage at a fraction of the standard token count and runtime.
Δ-Encoder processes motion vectors and residuals through two lightweight branches designed to extract and compress codec-domain information. The resulting motion and residual tokens are concatenated to form the Δ-tokens used for P-frames, providing an efficient representation that is projected to the RGB token space during pre-training.
Training Paradigm. First, the Δ-Encoder is pre-trained to align its output with the frozen vision encoder. The resulting features are aligned with ground-truth image tokens via a patch-wise MSE loss, enforcing spatially consistent alignment across patches. After pre-training, the Δ-Encoder is integrated into the VideoLM for end-to-end fine-tuning — the reference-conditioned branches from pre-training are dropped, so no RGB reference frames are processed for P-frames. This yields a substantial compute and memory reduction while keeping the standard instruction tuning objective unchanged.
Acknowledgements
We would like to thank (in alphabetical order): Isar Meijer and Krzysztof Waraksa from Microsoft for help with training pipeline setup; Kevin Qu, Tao Sun and Jianhao Zheng from Stanford for feedback at different stages of the project.
Citation
If you find our work useful, please consider citing:
@misc{cope_videolm,
title={CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling},
author={Sayan Deb Sarkar and Rémi Pautrat and Ondrej Miksik and Marc Pollefeys and Iro Armeni and Mahdi Rad and Mihai Dusmanu},
year={2026},
eprint={2602.13191},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2602.13191},
}