Engineering at HeyGen: Inside the team building AI video at scale
Video transcript: Motion design isn't dead. In fact, we think it's just starting to expand and AI will play a huge part in unlocking the possibility for more people to create video. You see, every new model comes with a report card, a table of scores against benchmarks. These tests measure how well a model solves hard, checkable problems like code and agentic work or math and science. But many people are pushing these models somewhere else that is much harder to quantify. Creative work, UI design, writing, video. There's no answer key for those, so people improve by trial and error. Our team set out to change that for video generated from code, a quantifiable way to measure how well a model makes video. Introducing the Code2Video Benchmark. I'm sure you're asking, well, who says a video is good? We started with people, two generated videos side by side, followed by simple questions. Does it hold your attention? Does it match the brief? Thousands of judgments, all made by humans. Then we trained a model on those picks. Show it two videos and a question, and it tells you which one people would choose and why. This model flickers. That one ignores the brief. And because the judge can score any two videos in milliseconds, models can train against it, not just be ranked by it. Over time, this benchmark will help models produce better videos from code. This is just the beginning. HyperFrames. Go make something.
How a Video Company Built the #1 Voice Model
Oct 26, 2026HeyGen let users choose between leading voice engines. By using real-world feedback and blind preference testing, HeyGen built a voice model that surpassed competitors in quality while reducing costs by millions.
HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos
Oct 14, 2026An open beach. A moving car. A busy factory floor. HeyGen Voice brings your voice into the picture, shaping the delivery and sound to fit the scene.
Video Podcast: Enforcing Continuity Across Independently Rendered Turns
Aug 24, 2026Each speaker’s stretch of dialogue in a Video Podcast is rendered independently. The challenge is making those clips behave as though they came from the same recording.
HeyGen x Google Cloud: Bringing Avatar IV to TPUs
Aug 13, 2026HeyGen is an AI video generation platform with industry-leading avatar models. Avatar IV, the diffusion stack behind our talking-head videos, runs on more than 18B parameters and is available via Web and API. Working with the Google Cloud AI-Ninja team, we brought Avatar IV to an eight-chip Trillium (v6e) host and made it 1.86× faster than our first working version. It now matches the performance of our 8×H100 production setup while being up to 25% more cost-efficient per minute of generated video.
How Temporal Powers Workflows at HeyGen
Aug 5, 2026How we moved from a custom job system to durable execution across a multi-cloud GPU fleet.
Reframe Context Engineering - Lessons from Building Video Agent(s) - Part 1
Jul 27, 2026Context engineering is very different when your agent's goal is multi-modal (e.g. videos). Here's what we learned shipping one.
One Model, Multiple Controllability: Composable Behavior LoRAs for Video Avatars
Jul 23, 2026The expensive part of multi-behavior avatar control is a dataset where every behavior co-occurs, labeled, at studio quality — it barely exists. We skip it: train each behavior as a separate LoRA on cheap single-attribute data, compose them at inference, and close the gap with a short fine-tune on only a little real overlap.
Driving Real-Time Actions from Avatar Speech: A Word-Clock Architecture for the HeyGen Realtime API
Jul 15, 2026We synchronize AI avatar speech with live browser actions using per-word timestamps emitted by the Avatar Realtime API. By anchoring clicks, highlights, and scrolling to the exact word being spoken, the system keeps actions aligned with natural speech instead of unreliable timers. The approach powers interactive AI agents that can browse the web, teach, review code, and respond live while maintaining accurate audiovisual synchronization.
Auto-Optim: A Blind Agent Matched Our Hard-Won 3x, With No human in the Loop
Jun 29, 2026We handed Claude Code a benchmark, a tamper-proof quality gate, and one rule file, then walked away. On a single H100 it made the Wan 2.2 video decoder 3× faster on its own. The same speedup cost us 105 hand-steered experiments the first time; this run took 18, blind.
