Jekyll2026-05-04T04:03:38+02:00https://spliew.github.io/feed.xmlSeng Pei Liewmachine learning, physicsSeng Pei LiewICLR 2025 ML papers2025-02-18T00:00:00+01:002025-02-18T00:00:00+01:00https://spliew.github.io/posts/2025/02/blog-post-11I am curating (mainly) ICLR ‘25 submitted papers related to hyperparameter tuning of large-scale training.

Title Summary
Scaling Optimal LR Across Token Horizons ${\rm LR} \propto N^{-0.23}T^{-0.32}$ (fixed batch size)
How Does Critical Batch Size Scale in Pre-training? ${\rm crit. BS} \propto T$ (fixed LR)
Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit Relations of BS, LR, and $T$ are complicated
How to set AdamW’s weight decay as you scale model and dataset size the “timescale” 1/(LR * WD) should be constant
]]>Seng Pei LiewICLR 2023 privacy/FL papers2023-04-07T00:00:00+02:002023-04-07T00:00:00+02:00https://spliew.github.io/posts/2023/04/blog-post-10I have curated and am beginning to read ICLR ‘23 papers related to privacy and federated learning. The list will be constantly updated with the paper summaries. Stay tuned!

Title Summary
Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learning  
FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning  
On the Importance and Applicability of Pre-Training for Federated Learning  
The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation  
Bias Propagation in Federated Learning  
Federated Neural Bandits  
Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models  
Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation  
FedExP: Speeding Up Federated Averaging via Extrapolation  
A Statistical Framework for Personalized Federated Learning and Estimation: Theory, Algorithms, and Privacy  
Federated Learning from Small Datasets  
Effective passive membership inference attacks in federated learning against overparameterized models  
Hyperparameter Optimization through Neural Network Partitioning  
EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data  
CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning  
Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach  
Machine Unlearning of Federated Clusters  
Federated Nearest Neighbor Machine Translation  
Faster federated optimization under second-order similarity  
PerFedMask: Personalized Federated Learning with Optimized Masking Vectors  
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning  
FedFA: Federated Feature Augmentation  
Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?  
Generalization Bounds for Federated Learning: Fast Rates, Unparticipating Clients and Unbounded Losses  
Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection  
Better Generative Replay for Continual Federated Learning  
SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication  
Personalized Federated Learning with Feature Alignment and Classifier Collaboration  
FedDAR: Federated Domain-Aware Representation Learning  
Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning  
FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy  
Instance-wise Batch Label Restoration via Gradients in Federated Learning  
DepthFL : Depthwise Federated Learning for Heterogeneous Clients  
Multimodal Federated Learning via Contrastive Representation Ensemble  
FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification  
Private Federated Learning Without a Trusted Server: Optimal Algorithms for Convex Losses  
Sparse Random Networks for Communication-Efficient Federated Learning  
Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity  
Single-shot General Hyper-parameter Optimization for Federated Learning  
Test-Time Robust Personalization for Federated Learning  
Towards Addressing Label Skews in One-Shot Federated Learning  
MocoSFL: enabling cross-client collaborative self-supervised learning  
Individual Privacy Accounting with Gaussian Differential Privacy  
Regression with Label Differential Privacy  
Synthetic Data Generation of Many-to-Many Datasets via Random Graph Generation  
Differentially Private Adaptive Optimization with Delayed Preconditioners  
Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping  
Statistical Theory of Differentially Private Marginal-based Data Synthesis Algorithms  
Distributed Differential Privacy in Multi-Armed Bandits  
Differentially Private $L_2$-Heavy Hitters in the Sliding Window Model  
Efficient Model Updates for Approximate Unlearning of Graph-Structured Data  
Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference  
MPCFORMER: FAST, PERFORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC  
Disparate Impact in Differential Privacy from Gradient Misalignment  
Easy Differentially Private Linear Regression  
Stochastic Differentially Private and Fair Learning  
]]>Seng Pei LiewICML 2022 privacy papers2022-07-26T00:00:00+02:002022-07-26T00:00:00+02:00https://spliew.github.io/posts/2022/07/blog-post-9I have curated and am beginning to read ICML ‘22 papers related to privacy and federated learning. The list will be constantly updated with the paper summaries. Stay tuned!

Title Summary
FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning  
Differentially Private Approximate Quantiles  
A Joint Exponential Mechanism For Differentially Private Top-k  
Transfer Learning In Differential Privacy’s Hybrid-Model  
Bounding Training Data Reconstruction in Private (Deep) Learning  
FriendlyCore: Practical Differentially Private Aggregation  
Public Data-Assisted Mirror Descent for Private Model Training  
Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data  
Selective Network Linearization for Efficient Private Inference  
Shuffle Private Linear Contextual Bandits  
Optimal Algorithms for Mean Estimation under Local Differential Privacy  
Task-aware Privacy Preservation for Multi-dimensional Data  
Differentially Private Coordinate Descent for Composite Empirical Risk Minimization  
Private Streaming SCO in l_p geometry with Applications in High Dimensional Online Decision Making  
Private optimization in the interpolation regime: faster rates and hardness results  
Private Adaptive Optimization with Side information  
The Fundamental Price of Secure Aggregation in Differentially Private Federated Learning  
Faster Privacy Accounting via Evolving Discretization Improved numerical accountant
The Poisson Binomial Mechanism for Unbiased Federated Learning with Secure Aggregation  
Private frequency estimation via projective geometry  
Deduplicating Training Data Mitigates Privacy Risks in Language Models  
Hermite Polynomial Features for Private Data Generation use hermite poly feature instead of random fourier feature for mmd-like generative modeling
Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy  
Differentially Private Community Detection for Stochastic Block Models  
Improved Regret for Differentially Private Exploration in Linear MDP  
Differentially Private Maximal Information Coefficients  
Privacy for Free: How does Dataset Condensation Help Privacy? see this thread
Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data  
Tight and Robust Private Mean Estimation with Few Users  
Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning  
]]>Seng Pei LiewAwesome list of DP resources2021-12-18T00:00:00+01:002021-12-18T00:00:00+01:00https://spliew.github.io/posts/2021/12/blog-post-8I am curating a list of resources for learning subjects related to differential privacy, federated learning and machine learning (subject to my preferences).

This list will be updated constantly. Stay tuned!

Blogs, articles

Videos

Conferences (with videos)

Lectures, tutorials

Other awesome lists

]]>Seng Pei LiewPrivacy papers in ICML 20212021-06-06T00:00:00+02:002021-06-06T00:00:00+02:00https://spliew.github.io/posts/2021/06/blog-post-7I have curated and am beginning to read ICML ‘21 papers related to privacy and federated learning. The list will be constantly updated with the paper summaries. Stay tuned!
Note that I wrote a simple script to scrape the links to the paper and the links may not be accurate.

Title Summary
Differentially Private Quantiles  
Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix  
Large Scale Private Learning via Low-rank Reparametrization  
Differentially Private Densest Subgraph Detection  
PAPRIKA: Private Online False Discovery Rate Control  
FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Analysis  
Personalized Federated Learning using Hypernetworks  
Federated Composite Optimization  
Exploiting Shared Representations for Personalized Federated Learning  
Oneshot Differentially Private Top-k Selection  
Data-Free Knowledge Distillation for Heterogeneous Federated Learning  
Privacy-Preserving Video Classification with Convolutional Neural Networks  
Federated Continual Learning with Weighted Inter-client Transfer  
Federated Deep AUC Maximization for Hetergeneous Data with a Constant Communication Complexity  
Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning  
Differentially Private Query Release Through Adaptive Projection  
A Framework for Private Matrix Analysis in Sliding Window Model  
Federated Learning of User Verification Models Without Sharing Embeddings  
Differentially Private Aggregation in the Shuffle Model: Almost Central Accuracy in Almost a Single Message  
Locally Private k-Means in One Round  
[Differentially-Private Clustering of Easy Instances]  
Differentially Private Sliced Wasserstein Distance privatize sliced W. distance by utilizing random projection techniques plus gaussian noises. Applied to domain adaptation and gen. models. Showed that generating faces is feasible
Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning  
Ditto: Fair and Robust Federated Learning Through Personalization simple regularization between the global and local terms
Differentially Private Correlation Clustering  
HEMET: A Homomorphic-Encryption-Friendly Privacy-Preserving Mobile Neural Network Architecture  
Private Alternating Least Squares: (Nearly) Optimal Privacy/Utility Trade-off for Matrix Completion  
Lossless Compression of Efficient Private Local Randomizers  
Differentially Private Bayesian Inference for Generalized Linear Models  
Heterogeneity for the Win: One-Shot Federated Clustering  
Private Stochastic Convex Optimization: Optimal Rates in L1 Geometry  
Practical and Private (Deep) Learning without Sampling or Shuffling  
DeepReDuce: ReLU Reduction for Fast Private Inference  
Privacy-Preserving Feature Selection with Secure Multiparty Computation  
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation  
Accuracy, Interpretability, and Differential Privacy via Explainable Boosting  
Debiasing Model Updates for Improving Personalized Federated Training  
Leveraging Public Data for Practical Private Query Release  
One for One, or All for All: Equilibria and Optimality of Collaboration in Federated Learning  
Learner-Private Online Convex Optimization  
CRFL: Certifiably Robust Federated Learning against Backdoor Attacks  
Federated Learning under Arbitrary Communication Patterns  
Private Adaptive Gradient Methods for Convex Optimization  
]]>Seng Pei LiewPrivacy papers in ICLR 20212021-05-07T00:00:00+02:002021-05-07T00:00:00+02:00https://spliew.github.io/posts/2021/05/blog-post-6I have curated and am beginning to read ICLR ‘21 papers related to privacy and federated learning. The list will be constantly updated with the paper summaries. Stay tuned!
Note that I wrote a simple script to scrape the links to the paper and the links may not be accurate.

Title Summary
Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms  
Adaptive Federated Optimization  
Information Laundering for Model Privacy  
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning  
Federated Learning Based on Dynamic Regularization  
CaPC Learning: Confidential and Private Collaborative Learning  
Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning  
Do not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning  
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization  
Private Image Reconstruction from System Side Channels Using Generative Models  
FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning  
Differentially Private Learning Needs Better Features (or Much More Data)  
FedMix: Approximation of Mixup under Mean Augmented Federated Learning  
HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients  
R-GAP: Recursive Gradient Attack on Privacy  
Private Post-GAN Boosting Re-weight using MWEM the sequence of learned generators and discriminators to increase performance after training.
Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification  
Personalized Federated Learning with First Order Model Optimization no global model. personalize via interaction with other clients.
]]>Seng Pei LiewPrivacy papers in AISTATS 20212021-04-03T00:00:00+02:002021-04-03T00:00:00+02:00https://spliew.github.io/posts/2021/04/blog-post-5I have curated and am beginning to read AISTATS ‘21 papers related to privacy. The list will be constantly updated with the paper summaries. Stay tuned!
Note that I wrote a simple script to scrape the ArXiv links to the paper and the links may not be accurate.

Title Summary
Revisiting Model-Agnostic Private Learning: Faster Rates and Active Learning  
Differentially Private Analysis on Graph Streams  
Differentially Private Online Submodular Maximization  
On the Privacy Properties of GAN-generated Samples  
Robust and Private Learning of Halfspaces  
DP-MERF: Differentially Private Mean Embeddings with RandomFeatures for Practical Privacy-preserving Data Generation  
Stability and Differential Privacy of Stochastic Gradient Descent for Pairwise Learning with Non-Smooth Loss  
No-Regret Algorithms for Private Gaussian Process Bandit Optimization  
Federated f-Differential Privacy  
Quantifying the Privacy Risks of Learning High-Dimensional Graphical Models  
Optimal query complexity for private sequential learning against eavesdropping  
Differentially Private Weighted Sampling  
Shuffled Model of Differential Privacy in Federated Learning  
Private optimization without constraint violations  
Evading the Curse of Dimensionality in Unconstrained Private GLMs  
Location Trace Privacy Under Conditional Priors  
Differentially Private Monotone Submodular Maximization Under Matroid and Knapsack Constraints  
Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFT  
]]>Seng Pei LiewPrivacy papers in NeurIPS 20202020-10-12T00:00:00+02:002020-10-12T00:00:00+02:00https://spliew.github.io/posts/2020/10/blog-post-4I have curated and am beginning to read NeurIPS ‘20 papers related to privacy. The list will be constantly updated with the paper summaries. Stay tuned!
Note that I wrote a simple script to scrape the ArXiv links to the paper and the links may not be accurate.

Title Summary
A Simple and Practical Algorithm for Private Multivariate Mean and Covariance Estimation  
The Discrete Gaussian for Differential Privacy  
Private Identity Testing for High-Dimensional Distributions  
Differentially-Private Federated Contextual Bandits  
Permute-and-Flip: A new mechanism for differentially-private selection  
Auditing Differentially Private Machine Learning: How Private is Private SGD? Introduce a method to measure the emperically achievable value of epsilon. Also introduce an algorithm of poisoning that is effective against SGD clipping
AutoPrivacy: Automated Layer-wise Parameter Selection for Secure Neural Network Inference  
Adversarially Robust Streaming Algorithms via Differential Privacy  
Locally Differentially Private (Contextual) Bandits Learning  
Locally private non-asymptotic testing of discrete distributions is faster using interactive mechanisms  
On the Equivalence between Online and Private Learnability beyond Binary Classification  
A Scalable Approach for Privacy-Preserving Collaborative Machine Learning  
Private Learning of Halfspaces: Simplifying the Construction and Reducing the Sample Complexity  
Synthetic Data Generators – Sequential and Private  
Smoothly Bounding User Contributions in Differential Privacy  
Learning from Mixtures of Private and Public Populations  
Smoothed Analysis of Online and Differentially Private Learning  
Privacy Amplification via Random Check-Ins Try to solve the problem of determining the population size when using central DP in FL
The Flajolet-Martin Sketch Itself Preserves Differential Privacy: Private Counting with Minimal Space  
Understanding Gradient Clipping in Private SGD: A Geometric Perspective  
Differentially Private Clustering: Tight Approximation Ratios  
A Computational Separation between Private Learning and Online Learning  
Instance-optimality in differential privacy via approximate inverse sensitivity mechanisms  
Improving Sparse Vector Technique with Renyi Differential Privacy  
Breaking the Communication-Privacy-Accuracy Trilemma  
Inverting Gradients - How easy is it to break privacy in federated learning? Show that FL without DP is vulnerable to reconstruction attack, at least in Computer Vision
GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators sanitize selectively (only the generator) and bounding sensitivity with wasserstein distance instead of clipping.
Optimal Private Median Estimation under Minimal Distributional Assumptions  
Towards practical differentially private causal graph discovery  
Learning discrete distributions: user vs item-level privacy  
Faster Differentially Private Samplers via Rényi Divergence Analysis of Discretized Langevin MCMC  
CryptoNAS: Private Inference on a ReLU Budget  
A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling  
]]>Seng Pei LiewPrivacy papers in NeurIPS 20192020-09-22T00:00:00+02:002020-09-22T00:00:00+02:00https://spliew.github.io/posts/2020/09/blog-post-2I have curated and am beginning to read NeurIPS ‘19 papers related to privacy. The list will be constantly updated with the paper summaries. Stay tuned!

Title Summary
Private Hypothesis Selection Given samples from an unknown probability distribution, select a distribution from some fixed set of candidates which is “close” to the unknown distribution in some appropriate distance measure.
Differentially Private Algorithms for Learning Mixtures of Separated Gaussians Learning the parameters of Gaussian mixture models. sample complexity is small and no a priori bounds on the parameters of the mixture components.
Average-Case Averages: Private Algorithms for Smooth Sensitivity and Mean Estimation  
Generalization in Generative Adversarial Networks: A Novel Perspective from Privacy Protection Show that DP implies generalization. but their concrete examples (lipschitz constraint etc.) did not show how DP is achieved (?).
Differentially Private Bayesian Linear Regression  
Minimax Optimal Estimation of Approximate Differential Privacy on Neighboring Databases  
Locally Private Gaussian Estimation Each of n users draws a single i.i.d. sample from an unknown Gaussian distribution, and the goal is to estimate the mean of this Gaussian distribution while satisfying local differential privacy for each user.
Capacity Bounded Differential Privacy Limit the capability of adversary, i.e., adversary is capable of performing only linear classification.
Practical Differentially Private Top-k Selection with Pay-what-you-get Composition  
Privacy-Preserving Classification of Personal Text Messages with Secure Multi-Party Computation  
Efficiently Estimating Erdos-Renyi Graphs with Node Differential Privacy  
Differentially Private Markov Chain Monte Carlo  
Differentially Private Bagging: Improved utility and cheaper privacy than subsample-and-aggregate  
Oblivious Sampling Algorithms for Private Data Analysis  
Differentially Private Anonymized Histograms  
Facility Location Problem in Differential Privacy Model Revisited  
Private Learning Implies Online Learning: An Efficient Reduction  
Online Learning via the Differential Privacy Lens  
Elliptical Perturbations for Differential Privacy  
Limits of Private Learning with Access to Public Data  
Private Testing of Distributions via Sample Permutations  
Private Stochastic Convex Optimization with Optimal Rates  
Privacy-Preserving Q-Learning with Functional Noise in Continuous Spaces  
Privacy Amplification by Mixing and Diffusion Mechanisms  
On Differentially Private Graph Sparsification and Applications  
An Algorithmic Framework For Differentially Private Data Analysis on Trusted Processors  
User-Specified Local Differential Privacy in Unconstrained Adaptive Online Learning  
Differentially Private Covariance Estimation  
Differentially Private Distributed Data Summarization under Covariate Shift  
Locally Private Learning without Interaction Requires Separation  
Differential Privacy Has Disparate Impact on Model Accuracy If the original model is unfair, the unfairness becomes worse once DP is applied.
]]>Seng Pei LiewSome old blog posts2020-09-21T00:00:00+02:002020-09-21T00:00:00+02:00https://spliew.github.io/posts/2020/09/blog-post-1You can find some of my old posts about particle physics here.

]]>Seng Pei Liew