Tutorial: Demystifying Optimization and Generalization of Deep Learning via Feature Learning Theory

Description

This tutorial aims to introduce a theoretical framework called feature learning that allows to elucidate the optimization and generalization behaviors of deep neural networks, including convolutional neural networks and transformers, trained with different optimization algorithms such as SGD, Adam, SAM, and label-noise SGD. This tutorial will also discuss the interactions between these optimizers and various learning paradigms, including supervised learning, contrastive learning, and generative models. By connecting practical training phenomena with recent advances in feature learning theory, the tutorial provides a unified perspective on how deep networks achieve effective generalization under diverse conditions. The content covers theoretical insights, empirical findings, and open challenges in this evolving field.