3D attractor set of recurrent ReLU network (courtesy of Dave Nowak)
00:00
Rob Nowak
1,095 posts
Rob Nowak
@rdnowak
Director of the Center for the Advancement of Progress
Ho-Chunk land
Joined April 2009
- Here’s my take on the “mathematical foundations” of machine learning and AI. These course notes cover the basics of statistical learning theory, optimization, and functional analysis. nowak.ece.wisc.edu/MFML.pdf
- My dad (86 years old today!) is really into fractals and art. He’s sitting in on my ML course on neural nets etc, and he decided to start experimenting with fractal images generated using ReLU nonlinearities.
- What kinds of functions do deep neural networks learn? Now we know! arxiv.org/abs/2105.03361 Our paper presents a new representer theorem for deep ReLU networks and provides theoretical insights into weight decay, sparsity, skip connections, and low-rank weight matrices.
- Workshop on the Theory of Overparameterized Machine Learning, topml.rice.edu
- Rahul Parhi and I wrote a tutorial style article to explain modern theory of deep learning and neural function spaces via elementary signal/image processing concepts (Fourier transform, Radon transform, L1 regularization, sparsity) arxiv.org/abs/2301.09554
- What kinds of functions do neural networks learn? I gave a lecture on the topic at the Joint Statistical Meetings (JSM) this week. Slides are online here:
- I am immensely honored to be named the Grace Wahba Professor of Data Science. Grace has been a tremendous inspiration and guiding force throughout my entire career. research.wisc.edu/faculty-receiv…
- 1/4 Interesting facts concerning neural network depth (and activation functions): Finite-width, single hidden-layer neural networks cannot approximate continuous functions
- Video is now available for my recent lecture on “What kinds of functions do neural networks learn?” youtu.be/Q8JbCqPFxs4
- New paper on label-efficient supervised finetuning of LLMs. We address the expensive prompt annotation cost by humans/proprietary LLMs, saving as much as 50% on FLAN V2. Paper: arxiv.org/abs/2401.06692 Work led by: @jifan_zhang @cloudwaysX @BhattGantavya @arnaved 1/
- We are pleased to announce plans for a special issue of Signal Processing Magazine focused on the mathematics of deep learning: signalprocessingsociety.org/blog/ieee-spm-… We look forward to your submissions!
- A wonderful, must-read article summarizing Grace Wahba’s amazing contributions to statistics and machine learning. imstat.org/2021/05/14/ims…
- Congratulations to @yinglun122, who successfully defended his PhD thesis "Interactive ML: From Theory to Scale". He'll be starting at UC-Riverside as an Assistant Prof this summer!
