Tell Me What Happened: Unifying Text-guided Video Completion
via Multimodal Masked Video Generation

Tsu-Jui Fu¹ Licheng Yu² Ning Zhang² Cheng-Yang Fu²
Jong-Chyi Su³ William Yang Wang¹ Sean Bell² ¹UC Santa Barbara ²Meta ³NEC Lab Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Abstract

Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction. We then propose Multimodal Masked Video Generation (MMVG) to address this TVC task. During training, MMVG discretizes the video frames into visual tokens and masks most of them to perform video completion from any time point. At inference time, a single MMVG model can address all 3 cases of TVC, including video prediction, rewind, and infilling, by applying corresponding masking conditions. We evaluate MMVG in various video scenarios, including egocentric, animation, and gaming. Extensive experimental results indicate that MMVG is effective in generating high-quality visual appearances with text guidance for TVC.

Qualitative Results

👇 press the tab for different tasks and datasets

Kitchen (128²)
First Frame	w/o Text	Text	TATS	Ours	GT
		open cupboard
		turn on tap
		put down cloth
		put plate on counter
		put down cup
		pour water in pot
		rinse plate
		open fridge
		place pan
		close lid

Flintstones (128²)
First Frame	w/o Text	Text	TATS	Ours	GT
		They are looking at each other while Fred is speaking.
		Barney is in the living room, talking.
		Wilma is in a car and talking about something.
		Fred is talking to Barney and turning his head.
		Betty is talking in the living room and nodding.
		Fred is walking across a room, talking to himself.
		Betty is quickly opening and closing a window blind.
		Fred is walking outside while he is looking behind.
		Wilma gestures with one hand while talking.
		Barney is in the car talking and laughing.

MUGEN (128²)
First Frame	w/o Text	Text	TATS	Ours	GT
		Mugen lands on a snail to crush it. It walks right to collect coins and turns left.
		Mugen runs from left to right, collects coin, jumps to hit face, and jumps to down.
		Mugen walks to the right across a platform. It picks up a gem and a coin before crushing a worm.
		Mugen jumps over a gear and then collects a coin.
		Mugen jumps and moves left. And then collects three coins and a gem.
		Mugen runs from right to left and it collect coins there. It saw a ladybug so it runs from left to right.
		Mugen jumps down and up to collect a coin. Then it jumps left from right to collect the coins.
		Mugen climbs up the ladder and moves to the right. It jumps again and gets killed.
		Mugen walks to the right, gets a gem, and then jumps up to a platform for coins.
		Mugen jumps down to the middle floor to knock the worm and jumps down to the ground floor.

Kitchen (128²)
Last Frame	Text	Ours	GT
	turn tap water off
	open oven
	put down pepper
	close drawer
	add olive oil

Flintstones (128²)
Last Frame	Text	Ours	GT
	Betty is in the living room reading the paper.
	Wilma is riding in the car and talking.
	Barney is talking at the window.
	Fred stands over the fence to have a glimpse.
	Wilma is standing in the kitchen. She turns her head then she speaks.

MUGEN (128²)
Last Frame	Text	Ours	GT
	Mugen runs from right to left and gets the coins.
	Mugen runs from left to right, collects some coins, and kills a snail. And it jumps down.
	Mugen jumps to stage. It runs from left to right, collects a coin, and jumps over a worm. Then it jumps up.
	Mugen jumps over a box and then towards a coin.
	Mugen jumps up and gets the coins.

Kitchen (128²)
First Frame	Last Frame	Text	Ours	GT
		take plate
		turn down hob
		take lid off
		grab salt
		put plates in sink

Flintstones (128²)
First Frame	Last Frame	Text	Ours	GT
		Barney is trying to sing in a living room.
		Wilma is talking and holds a finger over her lips.
		Fred and Barney speak out loud.
		Wilma looks back at Betty who is speaking then she turns her head.
		Fred and Wilma are dancing across a room.

MUGEN (128²)
First Frame	Last Frame	Text	Ours	GT
		Mugen runs to left. Then collects a coin and a gem.
		Mugen jumps up the stage. It runs from left to right and jumps on a worm.
		Mugen jumps down the ladder and jumps up. It collects a gem.
		Mugen climbs up a ladder. It jumps onto a stack of boxes, drops down, and is killed by a worm.
		Mugen climbs a ladder onto a platform and kills a worm. It then walks to the right and towards a gem.

Kitchen (128²)
K Frames	Text	K=1	K=2	K=3	K=4	GT
	open drawer
	put bowl in saucepan
	put glove back
	place pepper shaker
	rinse knife

Flintstones (128²)
K Frames	Text	K=1	K=2	K=3	K=4	GT
	Wilma is talking on the phone.
	Fred and Barney shake hands across the rock wall.
	Betty and Wilma are peeking their heads into a room.
	Betty and barney are dancing in a room.
	Fred angrily points at Wilma while he talks to her.

MUGEN (128²)
K Frames	Text	K=1	K=2	K=3	K=4	GT
	Mugen runs to the right and jumps to an upper level, where it collects a coin.
	Mugen walks to the right to get a gem. It then walks past the ladybug and jumps.
	Mugen runs to the right, jumps onto a platform, and collects coins.
	Mugen walks. And then it jumps.
	Mugen goes from the left and collects coins.

Flintstones (128²)
First Frame	Text 1	Output 1	Text 2	Output 2
	Fred speaks to* a phone and laughs.*		Fred hangs up* a phone and laughs.*
	Fred is holding a bone beside his head.		Fred is moving a bone up and down beside his head.
	Wilma and Betty are walking through a room.		Wilma and Betty are speaking in a room.

MUGEN (128²)
First Frame	Text 1	Output 1	Text 2	Output 2
	Mugen jumps from left to right* to an upper platform.*		Mugen runs right to left and collects a gem.
	Mugen runs to the right. It jumps down the ground.		Mugen runs to the right. It jumps landing on a face.
	Mugen jumps.		Mugen jumps and gets a gem and coin.

MUGEN (128²)
Last Frame	Text 1	Output 1	Text 2	Output 2
	Mugen walks to the right and then jumps over a mouse* to collect coins.*		Mugen walks to the right and then jumps* to collect coins.*
	Mugen jumps* and moves to left.*		Mugen jumps down from a ladder* and moves to left.*
	Mugen moves from right to left* and climbs onto a ladder.*		Mugen moves from left to right* and climbs onto a ladder.*

MUGEN (128²)
First Frame	Last Frame	Text 1	Output 1	Text 2	Output 2
		Mugen jumps down a ladder, collects coins, and then jumps down.		Mugen jumps down a ladder, collects coins, and then jumps up and down.
		Mugen collects coins. Then it runs from left to right.		Mugen collects coins and jumps on face. Then it runs from left to right.
		Mugen jumps to get the coin. It then keeps walking and drops off the platform.		Mugen jumps to get the coin. It then jumps over a gear* and drops off the platform.*

UCF-101 (128²)

BAIR (64²)

UCF-101 (128²)
Typing	Writing on Board	Knitting	Pull Up	Mixing

Front Crawl	Skiing	Yo Yo	Playing Guitar	Surfing

WebVid (384²)
wash hand	cut chicken with knife	boy writes in notebook	sailboat on horizon	cloudscape time-lapse

type on laptop keyboard	downtown city with traffic car	cook goulash soup	wind turbine	flame and wood

pour coffee into cup	walk outdoors on beach	high-speed railway link	busy city of times square	young team busy discusses

green sea turtle swims and relaxes	man trains on exercise machine	flags flatter on strong wind	beautiful girl swings	river flows under old stone bridge

rotates apple lollipop	sunset on the baltic sea	woman unveils curtain	elephant shakes his head	hand of man playing guitar

designer mixes paint	underwater of coral reef	drive on empty road	child rides a bike	shave his face in bathroom

TGIF (384²)
shove food in his mouth	tilt her head	hug each other	surf in the ocean	sing a song

perform in front of audiences	skateboard down the hill	move his lips	talk and smile	stick her tongue out

kiss each other	ice skating	snowboard on a mountain	walk	palm tree is blowing

cut a mini pizza	swing his club	slap their hands together	flying above the clouds	run along a track