Flowception

Tariq Berrada Ifriqi, John Nguyen, Karteek Alahari, Jakob Verbeek, Ricky T. Q. Chen

Meta FAIR

We present Flowception, a novel non-autoregressive and variable-length video generation framework. Flowception learns a probability path that interleaves discrete frame insertions with continuous frame denoising. Compared to autoregressive methods, Flowception alleviates error accumulation/drift as the frame insertion mechanism during sampling serves as an efficient compression mechanism to handle long-term context. Compared to full-sequence flows, our method reduces FLOPs for training three-fold, while also being more amenable to local attention variants, and allowing to learn the length of videos jointly with their content. Quantitative experimental results show improved FVD and VBench metrics over autoregressive and full-sequence baselines. Finally, by learning to insert and denoise frames in a sequence, Flowception seamlessly integrates different tasks such as image-to-video generation and video interpolation.

Flowception iteratively inserts and denoises frames, progressively increasing temporal resolution.

• Installation
• Project Structure
• Supported Features
  • Generation Tasks
  • Guidance Methods
  • Conditioning
  • Model Backbones
• Configuration
• Required Paths
• Training
  • Tutorial notebook
  • Trainers
  • Quick start — Toy dataset
  • Single node, multi-GPU
  • Multi-node (SLURM)
  • Available configs
  • Resume from checkpoint
  • Key training config
• Sampling / Inference
• Config Reference
• Citation
• Acknowledgements

Flowception unifies multiple video generation tasks through its frame insertion mechanism. By varying the type and number of context (clean, visible) frames, the method currently supports:

Set the trainer via SOLVER.TRAINER:

SOLVER:
  TRAINER: flowception              # for I2V
  # TRAINER: flowception_t2v        # for T2V
  # TRAINER: flowception_interpolate  # for interpolation

Video interpolation and image-to-video generation across different datasets.

The denoiser backbone is selected via MODEL.CONDITION. Supported families:

conda env create -f environment.yaml
conda activate flowception
pip install -e .

conda create -n flowception python=3.10 -y
conda activate flowception
pip install -e .

The checked-in environment.yaml provides a minimal Python 3.10 environment and installs the project with pip install -e .. Adjust the PyTorch/CUDA packages for your machine if needed.

Key dependencies: PyTorch, HuggingFace diffusers, accelerate, transformers, submitit, optional wandb, and yacs.

flowception/
├── main.py                      # Training entry point
├── launcher_with_accelerate.py  # Multi-node SLURM launcher
├── launch_i2v.sh                # Example SLURM launcher for LTX I2V
├── launch_t2v.sh                # Example SLURM launcher for LTX T2V
├── launch_interp.sh             # Example SLURM launcher for LTX interpolation
├── test_run.sh                  # Sanity-check training script
├── config/
│   └── default.py               # All config defaults (YACS)
├── configs/
│   └── flowception/             # Toy, I2V, T2V, and interpolation configs
├── engine/
│   ├── trainer.py               # Base Trainer (training loop, checkpointing, EMA)
│   ├── flowception.py           # Flowception trainer (image-to-video)
│   ├── flowception_interpolate.py # Flowception trainer (interpolation)
│   ├── flowception_t2v.py       # Flowception trainer (text-to-video)
│   └── utils.py                 # Training utilities (augmentations, FSDP helpers)
├── modules/
│   ├── get_model.py             # Model registry — maps MODEL.CONDITION → architecture
│   ├── conditioning.py          # Text/image conditioners (CLIP, LLaMA, DINOv2)
│   ├── vae.py                   # VAE loader (LTX, Cosmos, and toy identity)
│   ├── ltx_flowception.py       # LTX-Video transformer with Flowception extensions
│   ├── metadit_flowception.py   # DiT-based Flowception backbone
│   ├── attention.py             # Attention layers (self, cross, memory-efficient)
│   ├── rope.py                  # Rotary position embeddings
│   ├── diffusion/               # Denoisers and sigma schedules
│   └── flowception/             # Flowception sampling, alignment, losses, and schedulers
├── data/
│   ├── datasets/
│   │   ├── dataset.py           # Dataset registry and loading
│   │   ├── dataloader.py        # Dataloader utilities
│   │   └── video/               # Video dataset implementations
│   └── loaders/
│       └── samplers.py          # Distributed samplers
└── helpers/
    ├── lr_schedulers.py         # Learning rate schedulers
    ├── checkpoint.py            # Checkpoint utilities
    ├── ema.py                   # EMA update
    └── modeling.py              # Gaussian log-likelihood utilities

Flowception uses YACS for configuration. All defaults are in config/default.py. Override via:

1. YAML config file (recommended): --config configs/flowception/i2v/openvid_128_ltx.yaml
2. CLI overrides: --append KEY VALUE (e.g., --append SOLVER.LR 0.0001 SOLVER.BATCH_SIZE 16)

Before training or inference, you should set the dataset and checkpoint paths your chosen config actually uses. Not every dataset backend consumes the same keys.

The checked-in data/datasets/paths.yaml is a public path template. It contains only LOCAL entries and placeholder paths; edit those values or add your own cluster key, then set DATA.CLUSTER to that key.

Flowception configs use the OpenVid-1M video dataset. Download and extract it:

# Download from HuggingFace
# https://huggingface.co/datasets/nkp37/OpenVid-1M

# Expected directory layout after extraction:
# /mnt/data/datasets/OpenVid-1M/
# ├── data/train/OpenVid-1M.csv   # annotations (video, caption, motion score, ...)
# └── video/                       # extracted .mp4 files

Update OPENVID1M.LOCAL.ROOT in data/datasets/paths.yaml to point to your download location. The checked-in OpenVid configs also set DATA.DATASET: openvid1m_flowception.

Supported datasets and path sections are:

CUSTOM_SUBJECTNESS and CUSTOM_WEBDATASET are public placeholders for custom dataset formats. Users must provide their own data in the documented structure before using those loaders.

Set CUSTOM_SUBJECTNESS.LOCAL.ROOT to a directory of joblib-readable .pt shards. Each shard should load to a list of dictionaries:

[
    {
        "filepath": "/absolute/path/to/video.mp4",  # or "path"
        "description": "caption or prompt text",
        "motion_score": 4.2,
        "subjectness": {
            "subjectness": 0.7,
            "size_ratio": 0.25,
        },
    },
]

The loader filters on motion_score, subjectness.subjectness, and subjectness.size_ratio, then reads the video from filepath or path.

Set CUSTOM_WEBDATASET.LOCAL.ORIGINAL.IMG_ROOT to a WebDataset root containing one or more .tar shards:

IMG_ROOT/
  shard-000000.tar
  shard-000001.tar
  nested/
    shard-000002.tar

Each tar sample should provide jpg, json, and txt fields. The json payload must include width, height, and AESTHETIC_SCORE; txt is the fallback caption.

CAPTIONS_0, CAPTIONS_1, and ENTROPY should mirror IMG_ROOT by relative path. For every image tar path, the loader replaces the caption/entropy root and swaps .tar for .pt. Each .pt sidecar should be a pickle/joblib-readable dictionary keyed by the WebDataset sample __key__:

# CAPTIONS_0 or CAPTIONS_1 sidecar
{"sample_key": "recaptioned text"}

# ENTROPY sidecar
{"sample_key": 5.8}

Set MODEL.VAE.NAME to choose a VAE:

Flowception supports three training frameworks. Set via SOLVER.TRAINER:

For a self-contained walkthrough of the Flowception method on a toy setting (no GPU or dataset required), see tutorial/tutorial.ipynb. It covers the core sampling algorithm, frame insertion mechanism, and training objective end-to-end.

# I2V
python main.py -c configs/flowception/toy/toy_coloring_i2v.yaml -t -n toy_i2v

# T2V
python main.py -c configs/flowception/toy/toy_coloring_t2v.yaml -t -n toy_t2v

# Interpolation
python main.py -c configs/flowception/toy/toy_coloring_interpolate.yaml -t -n toy_interp

# I2V — LTX_AE, Flowception V1 backbone, 128px, OpenVid-1M
accelerate launch --num_processes 8 main.py \
    -c configs/flowception/i2v/openvid_i2v.yaml -t -n i2v_run

# I2V — LTX-2B backbone, 256px, OpenVid-1M
accelerate launch --num_processes 8 main.py \
    -c configs/flowception/i2v/openvid_ltx_i2v.yaml -t -n ltx_i2v

# T2V — LTX-2B backbone, 256px, OpenVid-1M
accelerate launch --num_processes 8 main.py \
    -c configs/flowception/t2v/openvid_ltx_t2v.yaml -t -n ltx_t2v

# Interpolation — LTX-2B backbone, 256px, OpenVid-1M
accelerate launch --num_processes 8 main.py \
    -c configs/flowception/interpolate/openvid_ltx_interpolate.yaml -t -n ltx_interp

# T2V — 4 nodes × 8 GPUs = 32 GPUs
python launcher_with_accelerate.py \
    --ngpus 8 --nodes 4 \
    --partition your_partition \
    --cluster your_cluster \
    -c configs/flowception/t2v/openvid_ltx_t2v.yaml \
    -t -n ltx_t2v_32gpu

# Interpolation — 4 nodes × 8 GPUs
python launcher_with_accelerate.py \
    --ngpus 8 --nodes 4 \
    --partition your_partition \
    --cluster your_cluster \
    -c configs/flowception/interpolate/openvid_ltx_interpolate.yaml \
    -t -n ltx_interp_32gpu

# I2V — 2 nodes × 8 GPUs
python launcher_with_accelerate.py \
    --ngpus 8 --nodes 2 \
    --partition your_partition \
    --cluster your_cluster \
    -c configs/flowception/i2v/openvid_i2v.yaml \
    -t -n i2v_16gpu

Launcher options: --timeout (minutes, default 4320), --qos (job priority), --mem_gb (CPU RAM per GPU), --job_name (SLURM job name).

For single-node SLURM runs, the checked-in helper scripts mirror these commands: launch_i2v.sh, launch_t2v.sh, launch_interp.sh, and test_run.sh.

configs/flowception/
├── toy/
│   ├── toy_coloring_i2v.yaml           # Toy 3×3, Flowception V1 Tiny, identity VAE
│   ├── toy_coloring_t2v.yaml           # Same, T2V trainer
│   ├── toy_coloring_interpolate.yaml   # Same, interpolation trainer
│   └── toy_coloring_tiny.yaml          # Minimal toy Flowception config
├── i2v/
│   ├── openvid_i2v.yaml                # Flowception V1, 128px, llama3p2+dino, OpenVid-1M
│   ├── openvid_128_ltx.yaml            # Alias-style 128px I2V config
│   └── openvid_ltx_i2v.yaml            # LTX-2B, 256px, t5_xxl, OpenVid-1M
├── t2v/
│   └── openvid_ltx_t2v.yaml           # LTX-2B, 256px, t5_xxl, SNR=3, OpenVid-1M
└── interpolate/
    ├── openvid_interpolate.yaml        # Flowception V1, 128px, OpenVid-1M
    └── openvid_ltx_interpolate.yaml    # LTX-2B, 256px, t5_xxl, SNR=3, OpenVid-1M

Training automatically resumes if a checkpoint exists at ./logs/<name>/checkpoint/. You can also resume from a specific directory:

accelerate launch --num_processes 8 main.py \
    --train --resume_from /path/to/checkpoint_dir \
    --config configs/flowception/i2v/openvid_128_ltx.yaml \
    --name my_experiment

Generate samples from a trained checkpoint:

accelerate launch --num_processes 8 main.py \
    --sample_only \
    --config configs/flowception/i2v/openvid_128_ltx.yaml \
    --name my_samples \
    --logdir ./logs \
    --append \
        MODEL.WEIGHTS /path/to/trained_ckpt.bin \
        EVAL.NUM_SAMPLES 256

@misc{ifriqi2026flowceptiontemporallyexpansiveflow,
    title={Flowception: Temporally Expansive Flow Matching for Video Generation}, 
    author={Tariq Berrada Ifriqi and John Nguyen and Karteek Alahari and Jakob Verbeek and Ricky T. Q. Chen},
    year={2026},
    eprint={2512.11438},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2512.11438}, 
}

We thank the Lightricks team for open-sourcing LTX-Video. The LTX-Video transformer and VAE are the backbone of our strongest models, and their open release made this work possible. If you use the LTX-based Flowception configs, please also cite their work:

@article{HaCohen2024LTXVideo,
  title={LTX-Video: Realtime Video Latent Diffusion},
  author={HaCohen, Yoav and Chiprut, Nisan and Brazowski, Benny and Shalem, Daniel and Moshe, Dudu and Richardson, Eitan and Levin, Evgeny and Shiran, Guy and Zabari, Nir and Gordon, Ori and others},
  journal={arXiv preprint arXiv:2501.05219},
  year={2024}
}

This repository is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC-BY-NC 4.0).

The code and materials are available for non-commercial research and educational use. Commercial use is not permitted under this license. See LICENSE for details.