VlogReward: Learning Multi-Dimensional
Evaluation for Vlog Editing

ICML 2026

Yexiang Liu^‡1,2,3, Wen Zhong³, Sijie Zhu³, Xin Gu³, Fan Chen³,
Junxian Duan^1,2, Jie Cao^1,2, Longyin Wen³, Zhenfang Chen^*3

(^‡ work done during the internship at ByteDance ^* corresponding authors)

¹ MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences
² University of Chinese Academy of Sciences
³ ByteDance Intelligent Creation

Paper GitHub Dataset

Overview of VlogReward. Different from regular video understanding tasks whose processes are often simple and separated, vlog editing involves more complicated and interwoven processes. To conduct a comprehensive assessment for each step and element of vlog editing, the evaluation criteria would be multi-layered and highly subjective, showcasing unique challenges for vlog reward modeling.

Abstract

The rapid rise of vlogs as a personalized storytelling medium has created a demand for automated systems to evaluate and refine vlog editing plans. However, vlog assessment is highly subjective and remains challenging due to a lack of standardized criteria, dataset and benchmark, and effective reward models. To address these challenges, we define a comprehensive vlog evaluation framework guided by professional vlog creators and product managers, establishing a taxonomy of six key dimensions, i.e., Creativity, Consistency, Concept Design, Cinematography, Narration, and Pacing. Subsequently, we curate a large-scale dataset of 100k vlog edits and a dedicated benchmark, VRMBench, to evaluate the vlog rewarding capabilities of Multimodal Large Language Models (MLLMs). Finally, we present VlogReward, a robust vlog reward model that can provide both fine-grained multi-dimensional scores and actionable feedback for iterative refinement. Technically, we enhance the Group Relative Policy Optimization (GRPO) framework by introducing an adjustable inter-group comparison reward, which mitigates the “direction blindness” issue of standard GRPO and enables the model to better distinguish varied-quality edits. VlogReward achieves state-of-the-art results that significantly outperform existing MLLMs, including GPT-5 and Gemini-3-Pro. We hope that our study can help vlog creators and foster automated vlog evaluation and refinement systems.

Six-Dimensional Evaluation Taxonomy

With the guidance of professional vlog creators and product managers, we consolidate the vlog evaluation metrics into six key dimensions and assign integer ratings from 1 to 5, i.e., k = 6 and s_k ∈ {1, 2, 3, 4, 5}.

Creativity

Narrative inventiveness and originality.

Consistency

Text–visual alignment.

Concept Design

Script element appropriateness.

Cinematography

Shot selection and framing.

Narration

Storyline and narration quality.

Pacing

Temporal rhythm and sequence logic.

Six-dimensional taxonomy. The rubric decomposes vlog editing quality into creative intent, semantic consistency, concept design, cinematography, narration, and pacing. Each candidate editing plan receives a 1-5 integer score on every dimension, so the evaluator can expose both the overall preference and the specific editing weakness behind that preference.

Data Collection Pipeline

Figure 2. Data collection pipeline. Subjective Dataset: We first collect raw assets v and use MLLMs to synthesize diverse editing plans (x₁, …, x_m) differing in subjective domains, e.g., aesthetics and creativity, then reward them to obtain multi-dimensional scores s and feedback f, with human verification to check for hallucinations and filter low-quality data. Objective Dataset: We additionally collect 8k assets, and progressively introduce tampering such as altering cut timestamps or subtitles to create objective factual errors, resulting in 4 plans of decreasing quality for each footage. Both splits are merged to form the final training dataset and the VRMBench benchmark.

Dataset & Benchmark

100k

Vlog edits (training)

400

Groups in VRMBench

Evaluation dimensions

1–5

Per-dimension rating

Figure 3. Data example. Raw video assets {v₁, …, v_n} are concatenated into a single video v, with each clip separated by a 3-second black screen. The editing plan x is a JSON encapsulating the cut timestamps, playback speed, voiceover, shot order, title, and subtitle. In this scheme, “shot timestamp” and “final timestamp” correspond to the time range in the stitched video v and the final edited vlog, respectively. The accompanying feedback provides scoring rationale, flags notable issues and improvement suggestions.

Figure 4. Vlog types. Our dataset contains various vlog types including daily life, recreation, hobbies, pets, sports, emotion, travel, and interview.

Figure 5. Data Statistics. On VRMBench, the footage durations are appropriately distributed, ranging from 10 seconds to 5 minutes. Both VRMBench-SB and VRMBench-OB contain balanced data across all score values, demonstrating the diversity of our dataset.

GRPO with Inter-Group Comparison Reward

Vanilla GRPO computes relative advantages only within intra-group rollouts and lacks cross-sample reference signals, leading to a “direction blindness” issue. We introduce an inter-group comparison reward while still computing relative advantages within intra-group rollouts, so that the model can better distinguish vlog edits of varying quality and more effectively identify the optimal editing plan. The final model uses \(m = 4\) candidate editing plans per asset, \(G = 8\) rollouts, and \(\lambda = 0.25\) unless otherwise stated.

Reward Function Setup

Format reward. The rollout must output integer scores from 1 to 5 and textual feedback for all six dimensions. If the required schema is satisfied, \(r_{\text{format}}\) is 1; otherwise it is 0, which masks out malformed answers in the total reward.

Score reward. For predicted scores \(s = \{s_1, \ldots, s_k\}\) and ground-truth scores \(\hat{s} = \{\hat{s}_1, \ldots, \hat{s}_k\}\) over \(k = 6\) dimensions, the reward is the exact-match ratio with a nominal correctly-formatted floor \(\alpha = 0.1\):

\[ r_{\text{score}} = (1 - \alpha)\frac{1}{k}\sum_{i=1}^{k}\mathbb{1}[s_i = \hat{s}_i] + \alpha . \]

Comparison reward. Let \(S_i^{[j]}\) be the total predicted score of rollout \(i\) for plan \(j\), and let \(\hat{S}^{[j]}\) be the ground-truth total score. For each alternative plan \(l\), the pairwise comparison reward is 1 when the predicted score difference has the same sign as the ground-truth quality difference, and 0 otherwise. The final \(r_{\text{comp}}\) averages these pairwise rewards over the other \(m - 1\) plans.

\[ r_{\text{comp}}(o_i^{[j]}, l) = \mathbb{1}\left[ \left(S_i^{[j]} - \frac{1}{G}\sum_{i'=1}^{G} S_{i'}^{[l]}\right) \left(\hat{S}^{[j]} - \hat{S}^{[l]}\right) > 0 \right], \] \[ r_{\text{comp}}(o_i^{[j]}) = \frac{1}{m - 1}\sum_{l \ne j} r_{\text{comp}}(o_i^{[j]}, l). \]

\[ r_{\text{total}} = r_{\text{format}}\left[(1 - \lambda)r_{\text{score}} + \lambda r_{\text{comp}}\right]. \]

Figure 6 (left). Illustration of the role of comparison reward (CR) beyond score reward (SR). In a simplified scenario with 2 rollouts and a single dimension, SR alone suffers from “direction blindness”, failing to recognize the preference direction. With SR + CR, the comparison signal provides an opposing policy update direction to restrain excessive updates of SR, maintaining score accuracy while ensuring score differences align with sample quality preferences.

Figure 7 (right). Comparison of reward scores of Qwen2.5-VL-7B and VlogReward. Plan 1 suffers from chronological disorder (day → night → day), while Plan 2 maintains a coherent temporal flow. Qwen assigns identical scores to both plans, failing to distinguish the edit quality; VlogReward accurately identifies the sequence-logic flaws in Plan 1 (assigning pacing score to 1).

Evaluation Results on VRMBench

Comprehensive evaluation of MLLMs and reward-model baselines on VRMBench. SB refers to VRMBench-SB, OB represents VRMBench-OB, and AVG indicates the average performance. VlogReward achieves the highest average performance across all metrics.

Model	Score Accuracy			Comparison Accuracy			Best-of-N Accuracy
Model	SB	OB	AVG	SB	OB	AVG	SB	OB	AVG
Closed-Source MLLMs
GPT-4o	24.4	32.4	28.4	38.1	51.8	44.9	24.3	42.1	33.2
GPT-5	29.7	43.2	36.4	53.7	72.5	63.1	37.4	65.7	51.5
o4-mini	24.3	38.2	31.3	46.1	64.6	55.3	26.5	47.7	37.1
Gemini-3-Pro	35.0	42.9	38.6	60.5	83.3	71.9	35.5	70.1	52.8
Open-Source MLLMs
LLaVA-OneVision-7B	24.0	26.7	25.4	40.9	44.8	42.8	18.9	21.9	20.4
LLaVA-NeXT-Video-7B	25.5	31.5	28.5	39.8	44.7	42.2	24.4	28.0	26.2
InternVL3.5-14B	23.2	29.5	26.3	30.2	36.8	33.5	20.3	35.1	27.7
Kimi-VL-A3B-Thinking-2506	22.9	30.4	26.7	41.6	44.8	43.2	22.7	23.6	23.1
MiMo-VL-7B-RL	22.5	31.4	26.9	44.4	53.9	49.2	28.5	39.1	33.8
Qwen2.5-VL-7B	22.4	32.9	27.7	34.8	42.8	38.8	20.5	31.9	26.2
Qwen2.5-VL-72B	26.1	38.1	32.1	36.8	55.3	46.0	24.8	44.5	34.6
Qwen3-VL-8B	22.8	31.6	27.2	39.3	54.8	47.0	24.9	47.0	35.9
Regressive Vlog Reward Models
RRM-MSE	23.6	25.4	24.5	42.3	45.5	43.9	31.6	41.0	36.3
RRM-BT	–	–	–	58.4	83.1	70.8	36.5	69.5	53.0
Generative Vlog Reward Models
GRM-SFT	37.4	41.4	39.4	54.7	65.0	59.8	40.3	44.1	42.2
GRM-GRPO	44.0	56.5	50.3	54.8	84.0	69.4	43.7	86.2	65.0
GRM-GRPO-Inter-Advantage	43.6	54.6	49.1	38.8	73.3	56.0	35.7	75.6	55.7
VlogReward (Ours)	45.9	56.7	51.3	60.4	86.6	73.5	48.3	86.8	67.6

Key Findings

General MLLMs remain deficient at vlog editing evaluation — most models cannot distinguish varied-quality editing plans, as evidenced by low Comparison and Best-of-N accuracies.
Objective vs. subjective gap: all models perform better on VRMBench-OB (factual errors) than on VRMBench-SB (aesthetic, subjective criteria).
Closed-source dominance: closed-source MLLMs generally outperform open-source counterparts; Gemini-3-Pro is the strongest baseline (71.9% Comparison, 52.8% Best-of-N).
Our VlogReward + CR achieves state-of-the-art results, improving Comparison Accuracy (69.4% → 73.5%), Best-of-N Accuracy (65.0% → 67.6%) and Score Accuracy (50.3% → 51.3%).

Ablation on λ and Comparison Reward

The coefficient λ controls the balance between exact per-dimension score matching and inter-plan discrimination. A small comparison weight improves both goals: at λ = 0.25, Score Accuracy reaches its best average value of 51.3, while Comparison Accuracy and Best-of-N Accuracy also improve over score-only GRPO. Larger values push the model toward stronger ranking behavior: λ = 0.75 obtains the best Comparison Accuracy (78.0) and λ = 1.0 obtains the best Best-of-N Accuracy (71.4), but pure comparison reward weakens absolute score calibration.

Metric (AVG)	λ = 0	λ = 0.25	λ = 0.5	λ = 0.75	λ = 1
Score Accuracy	50.3	51.3	50.1	48.8	20.3
Comparison Accuracy	69.4	73.5	75.0	78.0	77.9
Best-of-N Accuracy	65.0	67.6	71.0	71.3	71.4

Additional ablations show that the comparison reward is robust to smaller groups and fewer rollouts. With m = 3, G = 8, adding CR improves Score / Comparison / Best-of-N from 49.1 / 64.8 / 64.3 to 50.2 / 68.8 / 67.0. With m = 4, G = 4, CR still raises Comparison and Best-of-N from 67.9 / 65.3 to 72.0 / 66.8, even though fewer rollouts make score estimation slightly less stable. These results support the main design choice: CR should complement, rather than fully replace, score reward when both calibration and ranking are important.

BibTeX

If you find our work useful, please consider citing our paper:

@inproceedings{liu2026vlogreward,
      title={VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing},
      author={Liu, Yexiang and Zhong, Wen and Zhu, Sijie and Gu, Xin and Chen, Fan
              and Duan, Junxian and Cao, Jie and Wen, Longyin and Chen, Zhenfang},
      booktitle={Proceedings of the 43rd International Conference on Machine Learning},
      year={2026}
    }