That's a nice way of writing equations (I sometimes do this in lectures). From ICLR 2021 submission ("An attention free transformer"), openreview.net/forum?id=pW--c…
I passed this image through a pretrained ResNet 101 with @PyTorch and it predicts:
Tiger shark 23%
Hammerhead 21%
Great white shark 16%
Gar, garfish 11%
Sturgeon 3%
My conclusion : shape > [ texture + mountain context]
=> reassuring in some way.
(Image from @SolTight)
This Post is from an account that no longer exists. Learn more
This is on of the best papers I have recently read, congrats for this excellent work.
"NNs do not have an inherent “simplicity bias”. This property depends on components such as ReLUs, residual connections, and layer normalizations".
This also applies to transformers.
I trained a CNN w/o pooling on MNIST and produced a colored visualization of the feature maps you get when you translate a digit on an input canvas. This illustrates the equivariance property of convolutions.
More info (why convolutions?) in a blog post:
medium.com/@chriswolfvisi…
I love this figure showing the difference of manifolds learned by GANs and VAEs on a 1D toy example. The VAE learns the mean, the GAN passes through the data with higher fidelity but does not cover the full domain.
From: Plumerault et al. ICPR 2020
arxiv.org/abs/2012.11551
Integral neural networks look like a fantastic innovation: continuous representations along filter and channel dimension, prune w/o finetuning.
It is rare to discover a paper when the #CVPR proceedings are published, this paper was not on arxiv before..
openaccess.thecvf.com/content/CVPR20…
This new Biography of John von Neumann was an incredible read. I am not sure he was really human... The book is extremely well written and puts all his ideas and work into its historical context, eg the work of Turing and Gödel, and follow-up work, eg Conway etc.
A thread on similarities in the ways the field addressed two seemingly different problems, namely
A) recent work on transformers / self-attention addressing their quadratic complexity, and
B) work decoupling capacity and memory size in recurrent neural networks.
👇 1/19
< 2018: we have tried instance norm, batch norm, layer norm ...
2018: let's normalize over the missing dim, channel groups => ECCV 2018 best paper
< 2023: we applied the Fourier transform over 1d signals, images, time ....
2023: let's FFT channels => ICLR ratings of 8, 8, 10, 8.