David A. Snyder

Selected Publications

1. Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

   David Snyder, Apurva Badithela, Nikolai Matni, and 4 more authors

   Mar 2026

   Abs

   Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts. This strong resource constraint in real-world testing necessitates both more informative performance measures and reliable and efficient evaluation procedures to properly assess model capabilities and benchmark progress in the field. This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice. Based on safe, anytime-valid inference (SAVI), our test procedure is sequential, allowing the evaluator to stop early when sufficient statistical evidence has accumulated to reach a decision at a pre-specified level of confidence. Unlike previous work developed for binary success, our unified approach addresses a wide range of informative metrics: from discrete partial credit task progress to continuous measures of episodic reward or trajectory smoothness, spanning both parametric and nonparametric comparison problems. Through extensive validation on simulated and real-world evaluation data, we demonstrate up to 70% reduction in evaluation burden compared to standard batch methods and up to 50% reduction compared to state-of-the-art sequential procedures designed for binary outcomes, with no loss of statistical rigor. Notably, our empirical results show that competing policies can be separated more quickly when using fine-grained task progress than binary success metrics.

2. 

   Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators

   Apurva Badithela, David Snyder, Lihan Zha, and 4 more authors

   Oct 2025

   Abs Bib Website

   Rapid progress in imitation learning, foundation models, and large-scale datasets has led to robot manipulation policies that generalize to a wide-range of tasks and environments. However, rigorous evaluation of these policies remains a challenge. Typically in practice, robot policies are often evaluated on a small number of hardware trials without any statistical assurances. We present SureSim, a framework to augment large-scale simulation with relatively small-scale real-world testing to provide reliable inferences on the real-world performance of a policy. Our key idea is to formalize the problem of combining real and simulation evaluations as a prediction-powered inference problem, in which a small number of paired real and simulation evaluations are used to rectify bias in large-scale simulation. We then leverage non-asymptotic mean estimation algorithms to provide confidence intervals on mean policy performance. Using physics-based simulation, we evaluate both diffusion policy and multi-task fine-tuned }(π_0}) on a joint distribution of objects and initial conditions, and find that our approach saves over }(20-25}%}) of hardware evaluation effort to achieve similar bounds on policy performance.

     title = {Reliable and {Scalable} {Robot} {Policy} {Evaluation} with {Imperfect} {Simulators}},
     author = {Badithela, Apurva and Snyder, David and Zha, Lihan and Mikhail, Joseph and O'Kelly, Matthew and Dixit, Anushri and Majumdar, Anirudha},
     publisher = {arXiv},
     month = oct,
     year = {2025},
     note = {arXiv:2510.04354 [cs]},
     keywords = {Computer Science - Artificial Intelligence, Computer Science - Robotics, Electrical Engineering and Systems Science - Systems and Control},
     doi = {10.48550/arXiv.2510.04354},
   }

3. 

   Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping

   David Snyder, Asher James Hancock, Apurva Badithela, and 6 more authors

   In Proceedings of the Robotics: Science and Systems XVIII Conference (RSS 2022) Jun 2025

   Abs arXiv Bib Website

   Imitation learning has enabled robots to perform complex, long-horizon tasks in challenging dexterous manipulation settings. As new methods are developed, they must be rigorously evaluated and compared against corresponding baselines through repeated evaluation trials. However, policy comparison is fundamentally constrained by a small feasible sample size (e.g., 10 or 50) due to significant human effort and limited inference throughput of policies. This paper proposes a novel statistical framework for rigorously comparing two policies in the small sample size regime. Prior work in statistical policy comparison relies on batch testing, which requires a fixed, pre-determined number of trials and lacks flexibility in adapting the sample size to the observed evaluation data. Furthermore, extending the test with additional trials risks inducing inadvertent p-hacking, undermining statistical assurances. In contrast, our proposed statistical test is sequential, allowing researchers to decide whether or not to run more trials based on intermediate results. This adaptively tailors the number of trials to the difficulty of the underlying comparison, saving significant time and effort without sacrificing probabilistic correctness. Extensive numerical simulation and real-world robot manipulation experiments show that our test achieves near-optimal stopping, letting researchers stop evaluation and make a decision in a near-minimal number of trials. Specifically, it reduces the number of evaluation trials by up to 32% as compared to state-of-the-art baselines, while preserving the probabilistic correctness and statistical power of the comparison. Moreover, our method is strongest in the most challenging comparison instances (requiring the most evaluation trials); in a multi-task comparison scenario, we save the evaluator more than 160 simulation rollouts.

     title = {Is Your {Imitation} {Learning} {Policy} Better than Mine? {Policy} {Comparison} with {Near}-{Optimal} {Stopping}},
     author = {Snyder, David and Hancock, Asher James and Badithela, Apurva and Dixon, Emma and Miller, Patrick and Ambrus, Rares Andrei and Majumdar, Anirudha and Itkina, Masha and Nishimura, Haruki},
     booktitle = {Proceedings of the {Robotics}: {Science} and {Systems} XVIII {Conference} ({RSS} 2022)},
     volume = {21},
     shorttitle = {{Policy} {Comparison} with {Near}-{Optimal} {Stopping}},
     month = jun,
     year = {2025},
     isbn = {979-8-9902848-1-4},
   }

4. 

   Failure Prediction with Statistical Guarantees for Vision-Based Robot Control

   Alec Farid^*, David Snyder^*, Allen Z. Ren, and 1 more author

   In Proceedings of the Robotics: Science and Systems XVIII Conference (RSS 2022) Jun 2022

   Abs arXiv Bib Website

   We are motivated by the problem of performing failure prediction for safety-critical robotic systems with high-dimensional sensor observations (e.g., vision). Given access to a black-box control policy (e.g., in the form of a neural network) and a dataset of training environments, we present an approach for synthesizing a failure predictor with guaranteed bounds on false-positive and false-negative errors. In order to achieve this, we utilize techniques from Probably Approximately Correct (PAC)-Bayes generalization theory. In addition, we present novel class-conditional bounds that allow us to trade-off the relative rates of false-positive vs. false-negative errors. We propose algorithms that train failure predictors (that take as input the history of sensor observations) by minimizing our theoretical error bounds. We demonstrate the resulting approach using extensive simulation and hardware experiments for vision-based navigation with a drone and grasping objects with a robotic manipulator equipped with a wrist-mounted RGB-D camera. These experiments illustrate the ability of our approach to (1) provide strong bounds on failure prediction error rates (that closely match empirical error rates), and (2) improve safety by predicting failures.

     title = {{Failure} {Prediction} with {Statistical} {Guarantees} for {Vision}-{Based} {Robot} {Control}},
     author = {Farid<sup>*</sup>, Alec and Snyder<sup>*</sup></u>, <u>David and Ren, Allen Z. and Majumdar, Anirudha},
     booktitle = {Proceedings of the {Robotics}: {Science} and {Systems} XVIII {Conference} ({RSS} 2022)},
     volume = {18},
     month = jun,
     year = {2022},
     isbn = {978-0-9923747-8-5},
   }

5. 

   Generating Adversarial Disturbances for Controller Verification

   Udaya Ghai^*, David Snyder^*, Anirudha Majumdar, and 1 more author

   In Proceedings of the 3rd Conference on Learning for Dynamics and Control (L4DC 2021) May 2021

   Oral Presentation Abs arXiv Bib Website

   Selected for Oral Presentation (< 10%) at the 3rd Conference on Learning for Dynamics and Control (L4DC 2021)

   We consider the problem of generating maximally adversarial disturbances for a given controller assuming only blackbox access to it. We propose an online learning approach to this problem that adaptively generates disturbances based on control inputs chosen by the controller. The goal of the disturbance generator is to minimize regret versus a benchmark disturbance-generating policy class, i.e., to maximize the cost incurred by the controller as well as possible compared to the best possible disturbance generator in hindsight (chosen from a benchmark policy class). In the setting where the dynamics are linear and the costs are quadratic, we formulate our problem as an online trust region (OTR) problem with memory and present a new online learning algorithm (MOTR) for this problem. We prove that this method competes with the best disturbance generator in hindsight (chosen from a rich class of benchmark policies that includes linear-dynamical disturbance generating policies). We demonstrate our approach on two simulated examples: (i) synthetically generated linear systems, and (ii) generating wind disturbances for the popular PX4 controller in the AirSim simulator. On these examples, we demonstrate that our approach outperforms several baseline approaches (including H-infinity disturbance generation and gradient-based methods).

     title = {{Generating} {Adversarial} {Disturbances} for {Controller} {Verification}},
     author = {Ghai<sup>*</sup>, Udaya and Snyder<sup>*</sup></u>, <u>David and Majumdar, Anirudha and Hazan, Elad},
     booktitle = {Proceedings of the 3rd {Conference} on {Learning} for {Dynamics} and {Control} (L4DC 2021)},
     month = may,
     year = {2021},
     publisher = {PMLR},
     pages = {1192--1204},
     issn = {2640-3498},
   }

Patents

1. 

   Omnidirectional flow sensor (US Patent App. 18/367,015)

   Nathaniel Simon, Alexander Piqué, David Snyder, and 3 more authors

   2024

   Abs Bib Website

   Disclosed is a fast-response sensor system to measure a fluid vector in a plurality of dimensions. This system, known as ‘MAST’ (for MEMS Anemometry Sensing Tower), utilizes microelectromechanical (MEMS) hot-wire devices to produce a solid-state, lightweight, and robust flow sensor system suitable for, e.g., real-time wind estimation onboard a UAV. The MAST uses three through eighteen polygonally-arranged microscale flow sensors to determine the fluid’s vector’s direction and magnitude.

   @article{simon2024omnidirectional,
     title = {Omnidirectional flow sensor (US Patent App. 18/367,015)},
     author = {Simon, Nathaniel and Piqu{\'e}, Alexander and Snyder, David and Ikuma, Kyle and Majumdar, Anirudha and Hultmark, Marcus},
     year = {2024},
     publisher = {Google Patents},
     note = {US Patent App. 18/367,015},
     patents = {true},
   }

PhD Thesis

1. Nonasymptotic Methods for Guaranteed Robotic Policy Synthesis and Evaluation

   David Snyder

   2025

   Abs Bib

   In the past decade, robotics research has undergone an unremittingly Schumpeterian evolution, manifesting far-reaching and rapid progress throughout the field. Entire analytical paradigms, developed over decades, have been superseded in empirical performance by novel advances leveraging vast amounts of newly available computation and training data. As a result, current robots are capable of complex, dexterous behaviors which empirically generalize across significant variation in task settings and environments. However, not all is solved; robots which appear to be robust often demonstrate brittle and unpredictable behavior in otherwise benign settings, where small, task-irrelevant fluctuations derail desired system behavior. Without reliable assurances that these autonomous systems are themselves reliable, the potential for deployment to address real-world problems remains fundamentally limited. To further the goal of deploying robots with assured safety and reliability, this dissertation constructs rigorous, interpretable, and implementable guarantees for robotic systems which account for the practical constraints imposed by the empirical nature of the robotics discipline. These guarantees are directed towards two critical ends: (1) policy synthesis, or guarantees on a property of the robot’s decision-making rule; and (2) policy evaluation, or guarantees on a property of an evaluator’s decision-making process. The core challenge in robotics — and in synthesizing guarantees about robots — is the presence of significant, often-irreducible uncertainty in the operating environment. Robots must maintain rich sensor observations to interpret and interact with external objects, which themselves may be reasoning agents capable of adversarial (or collaborative) behaviors. The roboticist cannot arbitrarily reduce uncertainty by pre-filtering what the robot observes, and the representation of the robot’s uncertainty thus matters enormously. Under stochastic uncertainty, we focus on evaluation, including the development of provably near-optimal sequential procedures for comparing robot policies, and online failure predictors with guaranteed class-conditional correctness. Under nonstochastic uncertainty, we develop online learning procedures which construct robust disturbance generators and obstacle avoidance controllers. In each setting, guarantees are tailored to the small-data regime and enforce nonasymptotic (finite-sample) assurances. These methods represent important steps toward understanding the nature of uncertainty “in the robot’s world,’’ which is necessary for future deployment of safe, reliable robots capable of addressing practical problems.

   @phdthesis{snyder_nonasymptotic_2025,
     title = {Nonasymptotic Methods for Guaranteed Robotic Policy Synthesis and Evaluation},
     author = {Snyder, David},
     year = {2025},
     school = {Princeton University},
     thesis = {true},
   }