VideoPoet: A Large Language Model for Zero-Shot Video Generation
A multimodal large language model for zero-shot video and audio generation from images, videos, text, and audio.
About
Dan Kondratyuk
Email Google Scholar LinkedIn GitHub
I am currently a co-founder of rekursiv.ai, building teams of AI Scientists to autonomously create knowledge.
My work includes training foundational models at Luma AI and Google Research, along with efficient training and inference techniques for video models. At Luma AI, I co-developed the generative video models powering Dream Machine, and led work on realtime interactive video models.
I am the first author and lead contributor of VideoPoet, a multimodal LLM for video and audio generation that received an ICML Best Paper award. I also created MoViNets, a family of efficient mobile video networks for realtime video action recognition, and UDify, a single multilingual model for parsing the syntax of 75 languages.
At Luma AI, I led the World Models team, focused on realtime interactive GenAI video, including joint video-audio generation, video avatar speech-to-video, and realtime player and camera control. Before Luma, I spent five years at Google Research, starting as an AI Resident and later working as a Senior Machine Learning Engineer. My Google work included Integrated Multimodal Perception, a Mixture-of-Experts model that combines image, video, audio, and text modalities, as well as research on efficient model ensembling.
I hold a Joint M.S. in Computational Linguistics from Charles University in Prague and Saarland University, earned through an Erasmus Mundus scholarship. My thesis on multilingual dependency parsing received the Best Master’s Thesis award from the Mathematics & Physics department. I completed my B.S. in Computer Science at Boise State University, graduating Summa Cum Laude and receiving the Outstanding Student Award in Computer Science.
You can find my other accounts at the following links.
Selected Publications
My research spans multimodal generative models, efficient video understanding, and multilingual NLP. For a complete list, see my Google Scholar.
MoViNets: Mobile Video Networks for Efficient Video Recognition
A family of efficient mobile video networks enabling realtime action recognition on streaming video.
75 Languages, 1 Model: Parsing Universal Dependencies Universally
A multilingual multi-task model for universal part-of-speech tags, morphology, lemmas, and dependency trees across 75 languages.
Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
An overlooked approach showing that committees of models can achieve faster and more accurate predictions than single large models.
LemmaTag: Jointly Tagging and Lemmatizing for Morphologically Rich Languages with BRNNs
A featureless neural architecture for jointly generating part-of-speech tags and lemmas in morphologically rich languages.
Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception
A scalable multimodal multi-task model combining image, video, audio, and text modalities with alternating gradient descent and mixture-of-experts.
MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation
Memory-augmented latent transformers enabling consistent, high-quality video generation at arbitrary lengths.
CamViG: Camera Aware Image-to-Video Generation with Multimodal Transformers
Camera-aware image-to-video generation with multimodal transformers for controllable viewpoint synthesis.
