Andy Keller (@t_andy

Andy Keller

340 posts

Andy Keller

@t_andy_keller

Postdoctoral Fellow at The Kempner Institute at Harvard University -- Somewhere between Brains & Bits. PhD at UvA, Intern @ Apple MLR, Prev @ Intel AI & Nervana

AKAndyKeller.github.io

Joined March 2014

Pinned
Andy Keller
@t_andy_keller
Jul 22, 2025
Why do video models handle motion so poorly? It might be lack of motion equivariance. Very excited to introduce: Flow Equivariant RNNs (FERNNs), the first sequence models to respect symmetries over time. Paper: arxiv.org/abs/2507.14793 Blog: kempnerinstitute.harvard.edu/research/deepe… 1/🧵
GIF
58K
Andy Keller
@t_andy_keller
Mar 10, 2025
In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network? Super excited to share recent work with the brilliant @mozesjacobs: "Traveling Waves Integrate Spatial Information Through Time" 1/14
GIF
760K
Andy Keller
@t_andy_keller
Jul 26, 2023
Traveling waves are known to exist throughout the brain in a variety of forms — there are many hypotheses, but their exact computational role is debated. Together with @wellingmax we built an RNN which exhibits traveling waves to see what it could do. Here’s what we think: 1/7
GIF
122K
Andy Keller
@t_andy_keller
May 10, 2024
Traveling waves are indicative of conserved quantities. In the brain, there is undeniable evidence for traveling waves of neural activity -- but what is the brain trying to conserve? In our ICLR paper with @wellingmax, @_mullerlab, & @sejnowski, we ask: could it be memory? 🌊/9
GIF
76K
Andy Keller
@t_andy_keller
Sep 7, 2021
Together with @wellingmax, we think deep learning needs more organization and structure... topographic organization and equivariant structure 😁 Introducing our new paper: Topographic VAEs learn Equivariant Capsules 📃arxiv.org/abs/2109.01394 🧬github.com/AKAndykeller/T… 1/6
Andy Keller
@t_andy_keller
Nov 17, 2020
Excited to share my first paper! arxiv.org/abs/2011.07248 Self Normalizing Flows -- An efficient training method for unconstrained normalizing flows. Joint work w/ the ever supportive @jornpeters, @priyankjaini, @emiel_hoogeboom, Patrick Forré & @wellingmax 1/5
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
Just as ripples in water carry information across a pond, traveling waves of activity in the brain have long been hypothesized to carry information from one region of cortex to another*; but how can a neural network actually leverage this information?*cell.com/neuron/fulltex… 2/14
Traveling Waves in Visual Cortex
From cell.com
27K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
For all the technical details and more ablations, please see our paper recently accepted in workshop-form at ICLR Re-Align, and full pre-print on ArXiv. Code: github.com/KempnerInstitu… Paper: arxiv.org/abs/2502.06034 Hope to see you in Singapore! 🇸🇬 Fin/
GitHub - KempnerInstitute/traveling-waves-integrate: Repository to create traveling waves integrate...
From github.com
13K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
Inspired by Mark Kac’s famous question, "Can one hear the shape of a drum?" we thought: Maybe a neural network can use wave dynamics to integrate spatial information and effectively "hear" visual shapes... To test this, we tried feeding images of squares to a wave-based RNN: 3/14
GIF
27K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
We found that, in-line with theory, we could reliably predict the area of the drum analytically by looking at the fundamental frequency of oscillations of each neuron in our hidden state. But is this too simple? How much further can we take it if we add learnable parameters? 4/14
GIF
22K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
If you want more visualizations, a bit more depth, and even some audio of what different images 'sound' like to our models, please check out our @KempnerInst blog-post! kempnerinstitute.harvard.edu/research/deepe… 13/14
Traveling Waves Integrate Spatial Information Through Time - Kempner Institute
From kempnerinstitute.harvard.edu
12K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
We made wave dynamics flexible by adding learned damping and natural frequency encoders, allowing hidden state dynamics to adapt based on the input stimulus. On simple polygon images, we found the model learned to use these parameters to produce shape-specific wave dynamics: 6/14
GIF
22K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
Looking at the Fourier transform of the resulting neural oscillations at each point in the hidden state, we then saw that the model learned to produce different frequency spectra for each shape, meaning each neuron really was able to 'hear' which shape it was a part of! 7/14
17K
Andy Keller
@t_andy_keller
Mar 10, 2025
Replying to @t_andy_keller
Overall, we believe this is the first step of many towards creating neural networks with alternative methods of information integration, beyond those that we have currently such as network depth, bottlenecks, or all-to-all connectivity, like in Transformer self-attention. 12/14
13K