Latest Sessions
View All Sessions
S36 | Representation-Space MMD for Diffusion Language Models
Dmitry Baranchuk presents a post-training method for diffusion language models that minimizes the Maximum Mean Discrepancy between generated and reference distributions in the feature space of a frozen pretrained DLM. Keeping contextual features at individual token positions gives many observations per sequence from a single extractor pass, and the loss is optimized with policy gradients for discrete models or by differentiating through generated latents for continuous ones, with no full sampling trajectories or auxiliary models. The method lowers generative perplexity at comparable entropy on OpenWebText and increases decoding parallelism on 16B DMax-LLaDA2.0 models with similar or higher accuracy.
S35 | SOL: Measuring Gaps between Text Distributions
Gregor Kornhardt presents SOL, a distance between text distributions for evaluating diffusion and flow-based language models, which only provide likelihood bounds of varying tightness. SOL represents each sequence by the empirical measure of its hidden states under a fixed transformer and compares the distributions of these measures with the double sliced Wasserstein distance, which the authors prove is a metric when the transformer is injective. SOL detects distributional failures, recovers expected model trends, and gives stable sample-based estimates.
S34 | Simplex Diffusion Models
Justin Deschenaux presents Simplex Diffusion Models, which lift the diffusion process to the probability simplex so that beliefs over categories carry across denoising steps instead of collapsing through categorical sampling. SDMs have closed-form reverse transitions, train with a simple cross-entropy loss, and use a DDIM-like sampler with tunable stochasticity. They are competitive with strong discrete diffusion baselines on OpenWebText, beat masked and uniform diffusion on code generation, and, distilled to 8 steps, solve more GSM8K problems than distilled discrete diffusion models using 128 steps.
Featured Videos
View All Videos
How did diffusion LLMs get so fast?
Techniques for accelerating diffusion LLMs, from self-distillation and curriculum learning to KV caching and block diffusion
But How Do Diffusion Language Models Actually Work?
Jia-Bin Huang explores several ideas for applying diffusion models to language modeling
Simple Diffusion Language Models
Quick introduction to Masked Diffusion Language Models (MDLM) by Alexander Rush
About the Reading Group
Diffusion LLMs are faster, more controllable successors to traditional LLMs and are rapidly gaining adoption. This reading group builds a community for exchanging and debating emerging ideas in this space. While our primary focus is discrete diffusion models for language, we also welcome work on other modalities and applications, such as molecular design, drug discovery, and beyond.
Meet the Organizers
Subham Sahoo
Holds a Ph.D. from Cornell Tech, where he specialized in Diffusion Language Models. He has made foundational contributions to the field, with his work deployed at scale by Google, NVIDIA, and ByteDance across language generation and drug discovery.
Justin Deschenaux
PhD student in Machine Learning at EPFL, advised by Prof. Caglar Gulcehre. Previously interned at Apple MLR. His research interests include diffusion language models, fast generative models, and generalization.
