2nd Workshop on Deep Generative Model in Machine Learning:
Theory, Principle and Efficacy

Deep generative models have achieved remarkable empirical success, but their theoretical foundations remain poorly understood. This talk presents recent progress on the geometry and optimization of modern generative models, focusing on three representative settings. First, generative model inversion is analyzed with linear convergence of gradient descent established under two geometric conditions on the loss landscape, avoiding unrealistic random-weight assumptions. Second, transformer-based diffusion models trained on multi-token Gaussian mixture data are studied, showing that gradient descent converges to the Bayes-optimal denoiser and that self-attention approximates the optimal MMSE estimator. Third, Parsimonious Flow Matching (PFM) is introduced, which replaces the standard isotropic Gaussian latent with a multimodal mixture aligned with data structure, yielding better-conditioned optimization and faster convergence. Together, these results highlight how geometric structure in data and latent spaces enables sharper theoretical guarantees and more efficient generative modeling.

Compared to autoregressive models and even to continuous diffusions, diffusion language models offer a fundamentally different design space for crafting efficient and flexible generation processes. This talk discusses work along two axes of this design space: parallel decoding and variable-length generation. In the first half, an exact characterization of the optimal inference schedule for masked diffusion models is given, which depends on a certain "information profile" specific to the data distribution. From this characterization, simple schedules are derived that enable sampling provably more efficiently than autoregressive models for any distribution with bounded correlations. In the second half, FlexMDM is presented, a theoretically principled and empirically lightweight method for equipping diffusion language models with the ability to generate sequences of arbitrary length, while provably preserving their any-order generation capabilities.

In any generative model, the generated samples have a different distribution than the data distribution, due to inevitable learning errors. Moreover, this discrepancy, and metrics for evaluating the generated samples, are hard to characterize in high dimensions, motivating the need to understand how learning errors affect the reproducibility of certain "features" of the generated distributions. A first question is whether generative models produce "physical" samples, i.e., samples whose support is close to that of the true target distribution, despite algorithmic errors. A second question concerns what we term a lazy generative model: given samples from the target, we apply an arbitrary random dynamical system such that the distribution at finite time is approximately Gaussian. In principle, this noising process cannot be exactly inverted to recover target samples—but under what conditions can we approximately recover samples from a nearby distribution? The first part is joint work with Adriaan de Clercq (UChicago) and the second with Georg Gottwald (U Sydney).

We study Slowly Annealed Langevin Dynamics (SALD), a sampler for tracking a path of moving target distributions and approximating the terminal target through time slowdown. We establish non-asymptotic convergence guarantees via a KL differential inequality, showing that slowdown improves tracking through contraction of intermediate targets and a path-complexity term captured by an entropic action. Motivated by training-free guided generation with pretrained score-based generative models, we further introduce Velocity-Aware SALD (VA-SALD), which explicitly incorporates the underlying marginal distributions of the pretrained model and uses slowdown to correct the additional deviation induced by guidance. This yields a principled framework for training-free guided generation together with convergence guarantees that clarify the roles of intermediate functional inequalities, path complexity, and guidance bias.

Denoising diffusion models (DDMs) are state-of-the-art for numerous tasks pertaining to image generation and inverse problems. Yet many aspects of the training and sampling pipeline remain poorly understood. For instance, the necessity of noise conditioning has been particularly elusive, forcing practitioners to incorporate unnatural noise embeddings into neural network architectures and use ad hoc noise schedules during sampling. Motivated to remove these inconsistencies, a complete theory for blind denoising diffusion models (BDDMs) is provided: a variant of DDMs where the noise amplitude is not passed into the neural network during training nor sampling. The correctness of BDDMs as a sampling algorithm is justified under the sole assumption of low intrinsic dimensionality of the underlying data distribution relative to the ambient dimension. This assumption arises through the introduction of the Bayesian problem of estimating noise levels through a single noisy sample, which might be of independent interest. The performance of BDDMs is compared to standard DDMs in a variety of settings, showcasing the benefits of an adaptive scheme rigorously justified by theoretical analysis.

The goal of single-channel source separation is to reconstruct \(K\) sources given their mixture. In supervised settings where vast amounts of clean source data are available, this challenging, ill-posed problem has been addressed successfully by generative diffusion and flow-based prior models. However, access to such clean source samples is often limited. To bridge this gap, we present an unsupervised flow matching approach for source separation that learns directly from observed mixtures. This method relies on a novel combination of state-of-the-art supervised flow matching and regression-based self-supervised techniques. We provide insights into the objectives optimized by this approach and demonstrate it on image and audio benchmarks.

The aim of the talk is twofold. First, interesting noising processes besides Brownian motion are examined: the physics-inspired Kac process and the MMD gradient flow, leading to compactly supported measure curves and a better regularity of the flow matching velocity field. Second, a "quantile toolbox" for building generative models is presented: a unifying theory and a practical toolkit that turns latent noise selection into a data-driven design element.