Shapley values and related probabilistic values provide a flexible model-agnostic framework for attributing the behavior of a black-box system to data points, features, or other components, with a wide range of applications from model explanation and data valuation to credit assignment and compensation. Their practical use, however, is limited by a fundamental computational challenge: exact evaluation requires considering exponentially many coalitions, while even a single coalition evaluation may involve expensive model training or inference. In this talk, I will present a unified first-order perspective on Monte Carlo estimation of probabilistic values. Although existing estimators are motivated by seemingly different strategies, many share a common error structure that reveals how coalition sampling and surrogate modeling jointly determine statistical efficiency. This perspective leads to an Efficiency-Aware Surrogate-adjusted Estimator (EASE) that directly minimizes an explicit first-order approximation to the mean squared error. We demonstrate that EASE consistently improves estimation accuracy compared to existing estimators under the same utility-evaluation budget.

Transformer-based large language models are increasingly used for long-horizon tasks. Solving such tasks requires persistent memory rather than simply relying on an expensive and ever-growing context window. In this talk, I'll present our recent work on a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache. During "sleep", the model performs N offline recurrent passes over the accumulated context and updates the fast weights in its state-space model blocks. We show that increasing sleep duration N for our models improves performance, with the largest gains on examples that require deeper reasoning.

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at this http URL and installable via `pip install mup`.

Repeated snapshots of an evolving population reveal its marginal distributions at recorded times without identifying individual particle motion. A velocity field describes the local evolution of these distributions and permits interpolation at unobserved times. We study the tangent velocity field, defined as the compatible field of minimum instantaneous kinetic energy. For snapshots consisting of \(N\) fresh independent samples at each of \(M\) regularly spaced observation times, we establish the minimax estimation rate and construct a quadratic estimator that attains it. The rate is the larger of two terms: a temporal-grid term depending only on \(M\), reflecting the evolution left unobserved between observation times, and a statistical term depending on the total number \(MN\) of independent samples. We also establish an estimator-level counterpart of the classical convergence of rescaled optimal transport displacements to the tangent velocity field: a local semi-dual velocity estimator converges to the quadratic estimator in the small-displacement limit. For the trajectory experiment, which observes the same \(N\) particles at all \(M\) times along paths generated by the tangent velocity field, we derive the corresponding minimax rate. Neither experiment uniformly dominates across allocations \((M,N)\), while trajectories achieve a strictly faster rate when each experiment optimizes its allocation under a common recorded-position budget \(MN=\mathsf B\). We demonstrate a practical implementation that allows choosing any potential sieve whose functions and gradients can be evaluated at the observed positions. We select tuning parameters by cross-validation and use numerical examples to illustrate the two-term structure of the snapshot rate.

Model architectures are usually treated as fixed once pretraining is complete. This talk examines how pretrained transformers with full attention can instead be adapted into more efficient hybrid models without starting training over. Earlier conversion methods restore performance on short contexts but leave substantial gaps in long-context reasoning and retrieval. A recovery process combining representation alignment, off-policy distillation, and extended on-policy distillation addresses these gaps at a small fraction of the original pretraining cost. The results suggest that architectural decisions can remain open to revision well into a model’s training lifecycle.

We perform a mathematical and statistical analysis of the Wasserstein least squares problem, a regression method for vector-valued covariates and distribution-valued responses. Our proposal contrasts with other distributional regression methods by having a direct interpretation in terms of random variables, as a nonparametric analogue of the classic random-effects model. On the mathematical side, we use a strategy of Lavenant (2024) to show that Wasserstein least squares is the canonical extension of Euclidean least squares to the space of probability distributions from the perspective of convex analysis; this viewpoint gives rise to multimarginal and dual formulations of the Wasserstein least squares problem, extending a similar theory for Wasserstein barycenters. We perform a statistical analysis of the Wasserstein least squares problem under the template deformation model, showing, surprisingly, that estimation is possible at the n^{-1/2} rate. As a special case, we obtain improved rates of estimation for Wasserstein barycenters, which are an exponential improvement over those established by Ahidar-Coutrix, Le Gouic and Paris (2020). Finally, we propose a heuristic particle method for Wasserstein least squares and use it to conduct a novel analysis of large-scale demographic data from the RAND Health and Retirement Study.