Discrete Diffusion Reading Group

Samson Gourevitch (École Polytechnique), Yazid Janati (MBZUAI), and Dario Shariatian (INRIA) present their work revisiting Uniform Diffusion Models. Discrete diffusion models are usually trained by predicting clean tokens from a noisy sequence, but that prediction can be turned into reverse dynamics in several ways. For Masked Diffusion Models (MDMs) these choices largely coincide; for Uniform Diffusion Models (UDMs) they do not. This work shows that the standard plug-in parameterization for UDMs is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This exposes a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective. The authors characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score, which lets them pair either parameterization with either objective. The same conversions improve inference at no extra training cost, through an informed predictor-corrector sampler and temperature sampling applied to the leave-one-out predictor. The authors further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking step. On language modeling, leave-one-out parameterizations consistently improve UDM generation, and the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion comes less from the choice of marginals than from parameterization and sampling design.

Georgios Batzolis (University of Cambridge) presents Entropy-Gated Continuous Bitstream Diffusion. Diffusion language models promise parallel, order-agnostic generation, but on standard benchmarks they have long lagged behind autoregressive models in sample quality and diversity. Recent continuous flow and diffusion models over token embeddings narrowed this gap, suggesting that continuity itself is not the bottleneck. This work pushes the idea further by diffusing over bitstreams: each semantic token is encoded as a fixed-width sequence of binary bits, embedded in continuous space, and recovered from Gaussian noise by an EDM-style denoiser. Because an isolated bit has a known closed-form posterior under Gaussian corruption, the authors introduce a matched-filter residual parameterization, in which the network computes the analytic independent-bit posterior and spends its capacity only on the contextual residual. The largest gains come from the sampler. The deterministic probability-flow sampler is already competitive but over-contractive, reaching good perplexity by undershooting real-data token entropy. An entropy-gated stochastic sampler corrects this by applying Langevin-type churn on an entropy-rate grid, concentrating randomness in high-information regions and staying nearly deterministic elsewhere. On LM1B, a 130M-parameter model reaches a generative perplexity of 59.76 at matched data entropy using 256 function evaluations, surpassing prior diffusion baselines and the autoregressive reference. On OpenWebText it reaches 27.06 using four times fewer steps than prior 1024-step baselines. Because the model predicts O(log V) bitwise logits rather than a vocabulary-sized distribution, it also uses less memory and runs faster, an advantage that grows as vocabularies do.

Keya Hu and Linlu Qiu (MIT) present ELF (Embedded Language Flows). Diffusion and flow-based models are the default choice for generating continuous data such as images and video, yet today's leading diffusion language models still work over discrete tokens. ELF (Embedded Language Flows) asks whether continuous models can compete with only minimal special treatment of discreteness. ELF maps tokens into a continuous embedding space using an encoder needed only at training time, then learns a continuous-time Flow Matching process that denoises embeddings from Gaussian noise toward clean ones. It parameterizes the network to predict the clean embedding rather than the velocity, which lets a single shared-weight network handle both denoising and the final decoding step. Discretization happens only at the last step, where that same network projects the embedding through a learned unembedding matrix to token logits, so ELF needs no separate decoder at inference. Because the trajectory stays continuous almost everywhere, techniques from image diffusion such as classifier-free guidance carry over with little change. Across open-web text, machine translation, and summarization, ELF reaches lower generative perplexity than leading discrete models (MDLM, Duo) and concurrent continuous ones (FLM, LangFlow), using fewer sampling steps, roughly ten times fewer training tokens, and no distillation.