Generate Your Talking Avatar from Video Reference

Zujin Guo^1,2, Zhenhui Ye¹, Yi Ren¹, Yuanming Li¹, Ce Chen^1,3, Zhibin Hong¹, Chen Change Loy²

¹HeyGen Research, ²Nanyang Technological University, ³University of Melbourne

SIGGRAPH Asia 2026

TAVR replaces single-image avatar references with short video clips, enabling cross-scene generation with significantly better identity preservation. A three-stage training strategy bridges the domain gap between reference and target scenes. On a new cross-scene benchmark, TAVR yields the best identity similarity and achieves an overall quality score of 16.42 vs 14.13 for the next best method.

📖 For more visual results of TAVR, go checkout our project page.

---

• 📊 Cross-scene benchmark data is released: benchmark_data.json on Hugging Face!
• 🚀 Inference code is released!
• 🎉 TAVR has been accepted by SIGGRAPH Asia 2026!

Python 3.10, tested on one Hopper-class CUDA GPU with at least 80 GB of memory, and ffmpeg / ffprobe on PATH.

python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu126

flash_attn_3 has no PyPI wheel: build it from flash-attention (the hopper/ directory). FlashAttention 3 is the only attention backend.

1. Base models: Wan-AI/Wan2.1-T2V-14B (VAE, umT5) and facebook/wav2vec2-xlsr-53-espeak-cv-ft.
2. Detectors: yolo11x.pt (AGPL-3.0) and dw-ll_ucoco_384.onnx for whole-body pose (Apache-2.0).
3. TAVR checkpoint: the transformer .safetensors.

TAVR/
├── pretrained/
│   └── Wan2.1-T2V-14B/
│       ├── Wan2.1_VAE.pth
│       ├── models_t5_umt5-xxl-enc-bf16.pth
│       ├── google/umt5-xxl/
│       ├── wav2vec2-xlsr-53-espeak-cv-ft/
│       ├── yolo11x.pt
│       └── dw-ll_ucoco_384.onnx
└── ckpt/
    └── tavr_transformer.safetensors

One sample per directory:

TAVR/
└── samples/
    └── example1/
        ├── ref.mp4               # reference video of the person
        ├── target.png            # still of the target scene
        ├── target_caption.json   # {"caption": "..."} -- the positive prompt
        └── target.mp3            # driving audio (target.wav also works)

PYTHONPATH=. python infer.py \
  --sample-dir samples/example1 \
  --dit-ckpt ckpt/tavr_transformer.safetensors \
  --ckpt-dir . \
  --output-dir outputs

The result lands in outputs/example1/generated_target.mp4.

The cross-scene benchmark consists of 158 reference/target video pairs filtered from TalkVid. Its metadata is released as benchmark_data.json on Hugging Face:

hf download HeyGenAI/TAVR benchmark_data.json --local-dir ./benchmark

Each sample has:

The reference clip is the person's source video; the target clip provides the target still, the caption and the driving audio.

@inproceedings{guo2026generate,
     title={Generate Your Talking Avatar from Video Reference},
     author={Guo, Zujin and Ye, Zhenhui and Ren, Yi and Li, Yuanming and Chen, Ce and Hong, Zhibin and Loy, Chen Change},
      booktitle={SIGGRAPH Asia 2026 Conference Papers},
     year={2026}
}

Released under the Apache License 2.0. Third-party dependencies and their upstream attribution are listed in NOTICE. One of them carries non-permissive terms that apply at inference time and is not redistributed here: Ultralytics YOLO (AGPL-3.0).

TAVR builds on Wan2.1-T2V-14B and uses wav2vec 2.0, DWPose and Ultralytics YOLO. All videos and results shown here are for research demonstration purposes only.