MosaiChunk:Qualitative results
Prompt (excerpt)Full prompt
MosaiChunk preserves visual details across revisits. More in Video Viewer.
01 / INTRO
Long-horizon autoregressive video generation is limited by a finite context window. When an object or scene falls out of context, its fine-grained visual details may be lost and difficult to recover upon reappearance.
MosaiChunk composes a mosaic of selected historical key–value (KV) entries across space and time. A frozen video generator can directly consume such non-contiguous historical KV and recover the corresponding visual content. We therefore keep the generator fixed and learn only a lightweight router to preserve previously established visual details under a fixed active-memory budget.
02 / METHOD
Can a frozen backbone reuse a non-contiguous subset of historical KV?
To condition generation on more distant memory, existing retrieval methods select past chunks as additional context. But are whole chunks necessary?
To test whether a frozen video generator can consume a non-contiguous subset of KV entries, we select the KV at positions corresponding to the marked cookie region. Only the colored entries are supplied alongside the sliding window.
Memory architecture
Selecting non-contiguous historical KV entries for the relevant content can produce a consistent rollout. Our goal is to automatically identify and compose the right historical KV entries under a fixed active-cache budget.
1. Encode and store. Partition each chunk’s KV into equal-size sections. Encode a descriptor for each section and store both in CPU memory.
2. Retrieve. Use the latest chunk to query older memory. Score each historical section by its best descriptor match.
3. Compose. Select the highest-scoring sections within the budget. Feed their KV to the frozen generator alongside its sliding window.
Router details and equations
Stage 1: Encode and store. We partition each chunk's unrotated KV into equal-size groups, called sections, using balanced k-means. We observe that the KV entries for each section exhibit strong temporal redundancy: sections from neighboring chunks are always similar. Therefore, we train a lightweight descriptor encoder that maps the pooled keys of each section s to a descriptor d(s). The descriptor space learns an anti-recency bias so that similarity over descriptors reflects semantic relatedness rather than temporal proximity. The section bank stores these sections of verbatim KV entries and their descriptors in CPU memory.
Stage 2: Score historical sections. For retrieval when generating chunk cₜ, the sections of the latest generated chunk cₜ₋₁ form the query set Q, and the historical sections outside the sliding window form the candidate set H. The score measures a candidate's highest descriptor similarity to any query section.
Stage 3: Extract and compose. The router ranks all candidate sections by their scores and selects the global top-N. Since sections have equal size, the far-memory budget determines N. A softmax over all candidate scores gives a weight for each section. We normalize the selected weights to unit mean and use them to scale the selected sections' values. We then concatenate their KV into a MosaiChunk, which serves as far memory for chunk cₜ. The frozen DiT reads the MosaiChunk alongside the sliding window.
Self-distillation
Only the descriptor encoder is trained. A student using compact, routed memory learns to match the same frozen generator supplied with richer memory.
Training details
We use RAVEN-adapted MiniMax-H3 (H3-AR) for text-to-video (T2V) and LingBot-World-Infinity for image-to-video (I2V). We train the descriptor encoder through self-distillation to match the same frozen generator's predictions under richer far memory. Teacher and student use the same frozen DiT, noisy latent, denoising step, and sliding window; only far memory differs. The teacher receives whole historical chunks that fully cover the historical content to be redrawn. The student receives the router's MosaiChunk under a smaller memory budget. We minimize the mean squared error (MSE) between teacher and student predictions.
Teacher predictions are fixed targets. Gradients pass through the student's frozen DiT and the section value weights to the descriptor encoder; they do not pass through the discrete section selection. The stored KV and backbone parameters remain unchanged. Since the backbone itself supplies the target, training requires no rendered ground-truth target videos.
03 / BENCHMARK
RememBench tests whether a model remembers visual details after they leave its active context. Prompts drive revisits in T2V; camera motion drives them in I2V.
Objects appear, disappear, stay out of sight for five seconds, then return.
Model input: Prompt
1Appear
“wide open”
2Disappear
“closes completely”
3Hidden
“completely closed”
4Reappear
?“wide open again”
The object stays hidden longer than the recent context. We evaluate what returns.
Change the camera path. Return to the initial pose.
150 scenes · 50 indoor, 100 outdoorInitial pose Current poseEvery path returns to its starting position and orientation.
More T2V scenes
16 of 100 scenesThe same four-stage structure tests different objects and containers.
Model-generated frames when the prompt first reveals the object.
Benchmark protocol
The split contains 100 samples with scenarios disjoint from router training. Each model input extends Ring Forcing's three-stage appear–disappear–reappear design to four prompt segments, with a separate segment keeping the object out of sight. The nominal prompt transitions occur at 3.6, 6.2, and 11.2 seconds. The third segment therefore requests five seconds with the object out of sight, exceeding the largest sliding-window baseline's approximately three seconds of recent context.
04 / RESULTS
MosaiChunk improves revisit consistency over a sliding window (Base) and whole-chunk retrieval (MoC) under matched memory budgets.
Memory is measured in chunk equivalents. CLIP ↑ and LPIPS ↓ compare departure and revisit frames; scores are medians over scenes.
Revisit consistency vs. memory budget
CLIP ↑
Higher is betterLPIPS ↓
Lower is betterHover or focus a point for its method, memory budget, and score.
Better consistency with less memory. On T2V, MosaiChunk with 1 chunk of far memory achieves higher CLIP similarity than MoC with 2 chunks (0.899 vs. 0.841). On I2V rotation, MosaiChunk with 0.5 chunks achieves higher CLIP similarity than MoC with 2 chunks (0.815 vs. 0.807).
Evaluation details
Let F_c denote the far memory used to generate chunk c, and ‖F_c‖ its budget in video-chunk equivalents. For example, ‖F_c‖ = 1 means that the size of the far memory is equivalent to one video chunk. Retrieval methods keep their sliding window fixed, whereas Base matches the total active KV cache budget by retaining additional recent chunks.
CLIP similarity and LPIPS assess consistency between departure and revisit frames. T2V pairs are manually marked; I2V pairs use the conditioning frame and a revisit selected from Pi3X-reconstructed camera poses. We report medians over scenes.
BibTeX
@misc{zhang2026mosaichunkcompositingspatiotemporalmemory,
title = {MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation},
author = {Yiwen Zhang and Haocheng Xi and Michael Tian-Yue Liu and Alexei A. Efros and Hadar Averbuch-Elor and Qianqian Wang and Haiwen Feng},
year = {2026},
eprint = {2610.02153},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2610.02153},
}