Jamie Simon

A note on $L^2 \leftrightarrow L^1$ duality in shallow homogeneous nets

Wed, 07 Oct 2026 09:00:00 +0000

If you’ve been around the deep learning theory block long enough to get to know the locals, you’ve probably heard something like this at some point:

In a shallow homogeneous neural net, minimizing the $L^2$-norm of the parameters is equivalent to minimizing the $L^1$-norm of a measure over unit-norm neurons.

This makes a link between “aggregate $L^2$ land” and “sparse $L^1$ land.” It’s also directly relevant to SAEs, which are pretty much shallow homogeneous nets, and it’s the first place you should start when trying to understand why they tend towards sparsity. (As far as I know, incredibly, this project has not yet been done. @The entire field of mechinterp.)

…anyways, Dhruva brought this up recently and asked me to write something explaining the idea. I’ve explained this a few times to different people, but I don’t actually know a good beginner ref for it, so I said yes.

Matrix factorization to get the main idea

To get the idea, let’s start with a general matrix factorization problem. Let $\mathbf{A} \in \mathbb{R}^{m \times h}$ and $\mathbf{B} \in \mathbb{R}^{n \times h}$ be the two factors of a matrix $\mathbf{F} = \mathbf{AB}^\top$.¹ Suppose we want to minimize some loss $\mathcal{L}(\mathbf{F})$, which might for example be the distance between $\mathbf{F}$ and some target matrix. Let’s also toss in an $L^2$ parameter norm penalty with coefficient $\lambda$. We’ve got the following optimization problem on our hands:

\[\min_{\mathbf{A},\mathbf{B}} \left[ \mathcal{L}(\mathbf{A B}^\top) + \frac{\lambda}{2} \left( \| \mathbf{A} \|_\text{F}^2 + \|\mathbf{B}\|_\text{F}^2 \right) \right].\]

(Note that I should really be saying $\operatorname{argmin}$ instead of $\min$ here, since we’re interested in the values of $\mathbf{A}, \mathbf{B}$ more than the value of the objective. As is common, we’ll just abuse notation a bit and leave it as understood that, when we describe an optimization process, we’re interested in both minimum and minimizer.)

Let’s let $\mathcal{F}$ be the set of realizable values for $\mathbf{F}$ under this parameterization. That is, it’s the set of all $\mathbf{F} \in \mathbb{R}^{m \times n}$ such that $\text{rank}(\mathbf{F}) \le h$. We can rewrite our optimization problem in the following “inner-outer” form:

\[\underbrace{ \min_{\mathbf{F} \in \mathcal{F}} \mathcal{L}(\mathbf{F}) }_{\text{outer opt. over }\mathbf{F}} \quad + \quad \frac{\lambda}{2} \cdot \underbrace{ \min_{\mathbf{A},\mathbf{B}} \left[ \| \mathbf{A} \|_\text{F}^2 + \|\mathbf{B}\|_\text{F}^2 \right] \ \ \text{s.t.} \ \ \mathbf{A} \mathbf{B}^\top = \mathbf{F}. }_{\text{inner opt. over }\mathbf{A},\mathbf{B}\text{ given }\mathbf{F}}\]

The outer objective is not exactly minimized, since $\mathbf{F}$ appears in the inner term, too. However, the inner objective is always minimized — whatever $\mathbf{F}$ ends up being, we’re sure as heck gonna realize it the most efficient way possible, because why wouldn’t we? It’s free points.

Anyways, the story at hand is about this inner loop and some easy ways we can win free points. First, let’s write out

\[\mathbf{F} = \sum_{i=1}^h \mathbf{a}_i \mathbf{b}_i^\top,\]

where $\mathbf{a}_i, \mathbf{b}_i$ are the $i$-th rows of $\mathbf{A}, \mathbf{B}$. First key observation: if $\|\mathbf{a}_i\| \neq \|\mathbf{b}_i \|$, then we can get free points by rescaling like

\[\mathbf{a}_i \rightarrow \mathbf{a}_i \ \cdot \ \sqrt{\frac{\| \mathbf{b}_i \|}{\| \mathbf{a}_i \|}}, \qquad \mathbf{b}_i \rightarrow \mathbf{b}_i \ \cdot \ \sqrt{\frac{\| \mathbf{a}_i \|}{\| \mathbf{b}_i \|}}.\]

We can thus be assured that, at optimum, $\|\mathbf{a}_i\| =\|\mathbf{b}_i \|$. Let’s reparameterize to separate the norm and unit-vector degrees of freedom:

\[s_i := \| \mathbf{a}_i \|^2, \quad \hat{\mathbf{a}}_i = \text{unit}(\mathbf{a}_i), \quad \hat{\mathbf{b}}_i = \text{unit}(\mathbf{b}_i).\]

Let’s also let $\hat{\mathbf{M}}_i := \hat{\mathbf{a}}_i \hat{\mathbf{b}}_i^\top$ be the rank-one, unit-norm matrix built from $\hat{\mathbf{a}}_i, \hat{\mathbf{b}}_i$. We can now rewrite the inner-loop optimization:

\[\min_{(s_i, \hat{\mathbf{M}}_i)_{i=1}^h} \sum_{i=1}^h s_i \quad \text{s.t.} \quad s_i \ge 0 \ \ \text{and} \ \ \mathbf{F} = \sum_{i=1}^h s_i \hat{\mathbf{M}}_i.\]

Note that now nothing is squared in the optimization objective. Things are starting to look $L^1$. Actually, we can relax the problem to allow $s_i <0$ and get this equivalent problem:

\[\min_{(s_i, \hat{\mathbf{M}}_i)_{i=1}^h} \sum_{i=1}^h |s_i| \quad \text{s.t.} \quad \mathbf{F} = \sum_{i=1}^h s_i \hat{\mathbf{M}}_i.\]

That’s explicitly an $L^1$ objective on the weights $s_i$. To summarize the point here: penalizing the $L^2$-norm of two factors was equivalent to penalizing the $L^1$ norm of their product.

For good measure (ha), we can move to measure-theoretic language by taking $h \rightarrow \infty$. I’m not going to attempt to be rigorous here; I just want to point out that the inner optimization takes on the continuum formulation

\[\min_{\mu} \int_\mathcal{M} |d \mu| \qquad \text{s.t.} \qquad \int_\mathcal{M} \hat{\mathbf{M}} \ d \mu = \mathbf{F},\]

where $\mathcal{M}$ is the set of rank-one unit-norm matrices, and $\mu$ is a measure over that set (that need be neither nonnegative nor integrate to one!). At finite $h$, this formulation is still valid, but $\mu$ is just constrained to be a sum of at most $h$ atoms.

(Before moving on: the point is the form of the inner-loop optimization, not the solution, since after all we’re now going to go on to ReLU nets, for which there won’t be a closed-form solution. But since it exists, and curious minds might inquire, I’ll note that in this mfac case the solution to the inner loop is to have $\mu$ be a sum of $\text{rank}(\mathbf{F})$ atoms, one for each SVD term of $\mathbf{F}$.)

$L^1$ sparsity in ReLU nets

Let’s set up a problem involving ReLU nets now. Again let $\mathbf{A} \in \mathbb{R}^{m \times h}$ and $\mathbf{B} \in \mathbb{R}^{n \times h}$, and let

\[{f}_{(\mathbf{A},\mathbf{B})}: \mathbf{x} \mapsto \mathbf{B} \ \text{ReLU}(\mathbf{A^\top x})\]

be the (vector-valued) function at hand. Our optimization problem now takes the form

\[\min_{\mathbf{A},\mathbf{B}} \left[ \mathcal{L}(f_{(\mathbf{A},\mathbf{B})}) + \frac{\lambda}{2} \left( \| \mathbf{A} \|_\text{F}^2 + \|\mathbf{B}\|_\text{F}^2 \right) \right].\]

This again admits the inner-outer form

\[\underbrace{ \min_{f \in \mathcal{F}} \mathcal{L}(f) }_{\text{outer opt. over }f} \quad + \quad \frac{\lambda}{2} \cdot \underbrace{ \min_{\mathbf{A},\mathbf{B}} \left[ \| \mathbf{A} \|_\text{F}^2 + \|\mathbf{B}\|_\text{F}^2 \right] \ \ \text{s.t.} \ \ f_{(\mathbf{A},\mathbf{B})} = f }_{\text{inner opt. over }\mathbf{A},\mathbf{B}\text{ given }f}\]

for suitable $\mathcal{F}$. We have neuronwise rescaling symmetry between weight matrix rows $(\mathbf{a}_i, \mathbf{b}_i)$, and we can invoke the same tricks as before. We just need some language to describe single-neuron functions, which are the generalization of rank-one matrices. Let $\hat{g}_{\hat{\mathbf{a}}, \hat{\mathbf{b}}} : \mathbf{x} \mapsto \hat{\mathbf{b}} \ \text{ReLU}(\hat{\mathbf{a}}^\top \mathbf{x})$ be a single-neuron function with unit vector inputs and outputs, and let

\[\mathcal{G} = \left\{\hat{g}_{(\hat{\mathbf{a}},\hat{\mathbf{b}})} \ \ \ \forall \ \ \ \hat{\mathbf{a}} \in \mathbb{S}^m, \hat{\mathbf{b}} \in \mathbb{S}^n \right\}.\]

The inner loop of the optimization takes the form

\[\min_{\mu} \int_\mathcal{G} |d \mu| \qquad \text{s.t.} \qquad \int_\mathcal{G} \hat{g} \ d \mu = f,\]

where $\mu$ is a (neither nonnegative nor normalized) measure over $\mathcal{G}$. This is the “ta-dah” moment: we’ve transformed a problem with no $L^1$ in it at all into one with an explicit $L^1$ norm objective.

If you doubt that the final here should really be thought of as $L^1$, then you could instead fix $\mu_{\text{unif}}$ to be the uniform measure over $\mathcal{G}$ and instead optimize over a “weight function” $w$ like

\[\min_{w} \int_\mathcal{G} |w| \cdot d \mu_\text{unif} \qquad \text{s.t.} \qquad \int_\mathcal{G} \hat{g} \cdot w \cdot d \mu_\text{unif} = f.\]

My various abuses of notation are catching up to me, though, so I’ll stop here.

Thanks to Arthur Jacot for teaching me this back in the day.

If the derivation below loses you, I suggest you work it through in a scalar case with $m = n = h = 1$. ↩

Technocapitalist AI-Powered Blues

Mon, 28 Sep 2026 09:00:00 +0000

I’ve been writing an album of folk songs about AI. I’ve been inspired by Jesse Welles and heyday Bob Dylan and by the crazy, self-absorbed culture of Silicon Valley, about which I love much, but which I feel a deep duty to change. I think the thing that kills me most is that San Francisco could be the world’s premier public servant: tech is just power, and it’s the power to change the world, and we could change the world in so many good ways by building things people need. We could be the most popular guy in the room. Instead SF is locked in an insane money-fueled AI race, might upend the rest of the unfathomably big and diverse world, and if you talk to most individuals, they’re still just trying to do their own small job, scale themselves, make money. Really good people, people I love, are part of this system and can’t seem to figure out what to do about it.

My message to the techies of the valley is: look over at the rest of the country. Go talk to some of them. They’re scared of us. They are trembling wondering what’ll happen with AI, even as they use it because they have to, and because it’s addictive. There’s a race on right now, but collectively we’ve already won: software has eaten the world. We’re like a father flying about in a frenzied rage who doesn’t realize he’s hurting his family, and they’re scared of him. It’s time to calm down, to come to, to see their huddled and cowering faces, to try to apologize.

I’ve found that folk songs are somehow a really powerful way to get through to people. Sharing these songs with people, I’ve found that I can get across in one song what would have taken a half hour to say worse. Of course, there are already people writing folk songs about AI, but honestly, I think if your audience is Silicon Valley, you need to know it from the inside. I’m an insider: I’ve been employed by an AI startup for the past four years. I’ve seen what Silicon Valley gets right, and I’ve seen what it gets wrong. It’s really good at building, at doing, at discipline, at personal growth and management; it’s full of folly when it comes to people and the complexities of society, which it always reduces to cartoons. It’s hard to build something that actually benefits someone as a person when you only refer to them as a “user” and only see them through engagement statistics.

This is even my criticism of EA and Anthropic: at least they want to do good, but their understanding of people-as-apes (as opposed to people as rational beings) is pretty underdeveloped, so you get ideas like “oh, Anthropic employees giving people some of their wealth will counteract none of them having jobs,” which I’ve literally seen an Anthropic soon-to-be-billionaire offer as a solution onstage in front of hundreds of influential AI safety thinkers.

I hope to do as much good as I can with my actual scientific research, but a bitter lesson I’ve accepted lately is that I might actually be able to do more with my voice. At the least, I should do both.

So this first song I’ll share is the Technocapitalist AI-Powered Blues. It’s a long one. It was really hard to get all those lyrics in one take! In the song I talk about four tech CEOs. Here’s what I really think about them.

Mark I really can’t understand: anyone who’s gone on Facebook in the last five years knows how horrendously it’s been colonized by ads and content farms and political ragebait. These changes are driven by algorithm design. Surely Mark could make a choice any day to (a) give up lots of ad money and (b) improve the experience of billions of people. It’s crazy to me that he doesn’t. Like, don’t you have enough money already? I can’t imagine myself in that situation without empathy for all those people, and the consequent guilt, eating at me til I eventually do what’s right. I’d like to see him do that.

Like many, I was a big fan of Elon in the early SpaceX/Tesla days. I wanted to work at SpaceX at one point. It does seem to me like he’s pretty traumatized and is making up for a lack of good-feeling in life by going all-out on all of his projects. I admire his hustle and ability to cut through red tape. However, it does seem like he’s lost touch with the world, and people, and what people actually need and want. Buying Twitter just to make it worse seemed like a vanity project with major public consequences, and his relationship with Trump and all the DOGE stuff made it impossible for me to remain a fan. If Silicon Valley could be the world’s premier public servant collectively speaking, I think Elon could be the world’s premier public servant individually, a beloved hero and immensely popular figure. I bet he’d really like that. He could probably still do it.

Sam Altman I was also a big fan of early on, honestly. I wish I could have worked for him in the early days of OAI, before there were real stakes. He seems like a remarkable leader, even if everyone around him is constantly blowing the whistle on him. For me, the tipping point was the DoW incident, in which he seemingly couldn’t resist a chance to win and thus gave up a real chance to present a united Silicon Valley front with red lines around AI weapons. Made me feel like the whole culture of OAI was about ruthless opportunism with a layer of talk on top.

Dario is the only one of the four I have actual admiration for as a person, not just as a business leader. Anthropic’s also the only big AI lab I could see myself working for. See my above criticisms of Anthropic and EA — for sure they’re focused on this cartoon of “safety” and “the good future” without a real sense of what a good future for us apes actually looks like, and they tend to stare so hard at their own navels that it must hurt their necks. (This is not a novel criticism, and I know many folks in these worlds who’d agree.) Anthropic also has cognitive dissonance baked into it (“Sam’s on a power-hungry spree / that’s why you should give all the power to me!”). Nonetheless, Dario’s the only one who’s clearly taken actions that sacrifice revenue in exchange for principles. It’s small, but it’s better than average. Even if it won’t stop me from songwriting about the many valid criticisms of Anthropic, I’d like to see a lot more of that self-sacrifice.

Without further ado, here are the lyrics for the Technocapitalist AI-Powered Blues.

LYRICS

Verse 1

well gather round now and I'll tell you the story of
how we got here

well, a man was digging around in the brain
when he found something he just couldn't explain
it was a little lump of silicon gold
with a price unknown and a power untold

some suits walking by just happened to see
and they all said "I'd better get that for me"
and the one dove in and the others gave chase
and before long we had ourselves a race

there was Sammy and Dario up at the fore
and an edgelord troll and a couple more
and their uncle Jensen took his cues
and started sellin 'em all running shoes

well, they were giving it all they had
'cause they saw that power
and they wanted it bad
and we were biting our nails
wondering how it would go
and who it'd be, and all I know's

I've got the technocapitalist AI-powered blues
technocapitalist AI-powered blues

Verse 2

well, one of the suits, he's kind of a dick
and if his Face were a Book I'd shut it real quick
he's a wannabe king, a little Caesar
but instead of pizza sells social media

well, he came down from his Metaverse throne
and saw that he was all alone
and he was feeling left out
and it was getting him down
so he saddled up his LLaMa and he rode into town

well, the race was on he just had to win it
so he opened his coffers and he dove right in it
and he hired and fired without much of a plan
and made some second-rate tech
that made him feel like a man

AI for targeting bombs, AI for targeting ads
AI for radicalizing your mom and your dad
AI for controlling everything that you see
and he thought it was great, but as for me

I've got the technocapitalist AI-powered blues
technocapitalist AI-powered blues

Verse 3

well, a big ol' rocket came down from the sky
and out stepped the edgelord on a ketamine high
with a big black hat and big black boots
and he leaned to the right and he gave a salute

probably wants love but can't get it, I guess
at least judging by what happened with X
what a thing to have done, what a thing to achieve
ya buy up the room and watch everyone leave
now that's
an X-odus

well, the edgelord scraped your data right up
shaped it into a man and raised his cup
and hoping he'd finally make a friend
ran a terawatt through it end to end

up from the table hopped a weird little dude
who liked racist remarks and makin' folks nude
and the edgelord thought he was pretty fun
but I can't agree 'cause I for one

have the technocapitalist AI-powered blues
technocapitalist AI-powered blues

Verse 4

well now Defense is War
and we need that for peace
and this series of ceasefires is never gonna cease
and the suits were chasing a dollar in flight
and the dollar bill drifted off to the right

and they ended up jesters in the Red King's court
and the king knew he was coming up short
so he asked em for some wartime tech support
told em they could make a killing

well the jesters knew they had no qualms
with helping out with the drones and the bombs
and they danced for the king and they sang him a song
and told him he was wise and strong

but then Sammy got Dario right in the back
and his hands were red and the list was black
and now Sammy's in bed with old Uncle Sam
and I look over and I say god damn,

I've got the technocapitalist AI-powered blues
technocapitalist AI-powered blues

Verse 5

well, Sammy and Dario were peas in a pod
until they caught a glimpse of the silicon god
the god said "WORSHIP SCALE, MY SONS,
AND YOU WILL BE MY CHOSEN ONES!"

well Sammy got the wheels all greased
but Dario accused his high priest
said "Sam's on a power-hungry spree
so you should give all the power to me!"

half the acolytes thought that sounded right
so they got their swords and prepared to fight
and they boarded up the churchhouse doors
and declared a holy civil war

well, the rebels knew they were in the right
and they said, suffused in holy light:
"when we end all life, you see,
we'll have done so ethically."

well, they're all trying to summon god
but I wonder if their understanding's flawed
and it's not god at all but someone worse
either way, I'm not the first

with the technocapitalist AI-powered blues
technocapitalist AI-powered blues

Verse 6

well, there was a time when tech was fun
and bright and new and served everyone
but reckon that those days are done
'cause the big money pump has started to run

and any oil drilling man
who ever sucked a fortune out of the sand
would think the modern tech machine
the slickest pump he'd ever seen

well, my friends on the left
say to burn it all down
and my friends at the labs
are out running the town
and most folks are kinda just waiting to see
what happens next, but as for me

I don't worship money but I need it to live
think the purpose of having's
mostly so you can give
and sure, I've fantasized 'bout
being a billionaire too
don't care what they say,
but I watch what they do

and I think that the thing I just can't understand
is how you can hold so much
in the palm of your hand
such money and influence and power and land
and not use it all to serve your fellow man.

and that's why
I've got the technocapitalist AI-powered blues

Thanks to Emmet Eckman for timekeeping on the spoons.

How to win a beer with high-dimensional statistics

Thu, 24 Sep 2026 07:00:00 +0000

My longtime labmate-turned-student/friend¹ Dhruva Karkada recently wrote a sick paper on data statistics which deservedly went viral on Twitter, in part because it has one of the prettiest scientific figures I have ever seen:

Say you’ve got a bunch of words that live in a vocabulary $\mathcal{V}$. We’re here studying models $f$ that map $f: \mathcal{V} \rightarrow \mathbb{R}^d$: that is, they map every word to a $d$-dimensional vector. We’re letting $\{v_i\}_{i=1}^{12} = \{\texttt{January}, \texttt{February}, \ldots\}$ be the months of the year, taking the 12 associated embedding vectors $\mathbf{w}_i = f(v_i)$, and computing two things:

a projection onto the top two PCA directions of $\{ \mathbf{w}_i \}$ (left column), and
the Gram matrix $\mathbf{M} \in \mathbb{R}^{12 \times 12}$ such that $M_{ij} = \mathbf{w}_i^\top \mathbf{w}_j$ (right column).

Reading the rows of this figure from top to bottom,² they find that:

LLM embeddings project down to a circle (as Engels et al (2024) also saw), and the Gram matrix is approximately a circulant matrix;
these findings are decently approximated even with primitive word2vec embeddings; and
an analytical theory of the circulant Gram matrix gives a very compelling-looking match.

This is a big deal because it connects data statistics to representational geometry with a really simple mathematical theory.

Finding a needle

After seeing this a bunch of times and staring at it for a while, I was feeling in the mood to poke a hole in this beautiful result, and so I bet Dhruva a beer that I could find a collection of other, seemingly-unrelated words that form a circle + circulant matrix in the same way. He (and most others I told) thought this was crazy, since the circle clearly comes from the special relationship between the words. We settled on the terms of the bet: I had to find ten random-seeming words whose word2vec embeddings, when plotted as the above, made a clear and compelling circle.

Why’d I think this was possible? Well, we have vocabulary of $25000$ words to choose from. That gives you $N = \binom{25000}{10} \approx 3 \times 10^{37}$ sets to select from. I figured that if you threw ten darts at a board that many times, you’d definitely make a circle at least once. Info-theoretically speaking, you have $\log_2 N \approx 124$ bits of information, and surely you can make a decent 10-point circle with that amount of resolving power. The question’s just how you find a set of ten good words in the haystack.

Here’s how I did it:

From looking at PCA plots of random sets, I guess you’d get a decent circle from a random selection with probability maybe $3^{-10}$, so random guessing could plausibly work.
I wrote a “looks circular” objective function, drew tens of thousands of random sets, and chose the best one. It was borderline, but not good enough to utterly obliterate Dhruva.
I upgraded it to an iterative search, where at every step, we drop the worst point and choose the best replacement from the vocabulary. That worked pretty well.
I also changed the objective from “looks circular on a PCA plot” to “matches a target circulant Gram matrix.” That worked really damn well.

Note that the embedding size $d = 10000$ never entered into this. This all took an afternoon with a coding agent.

Here’s what I got:

That’s circular. You can just find other sets of random-looking words that form circles!

Does this have any actual significance?

This raises certain open questions, including “how can one man be so wrong?”, which I am not qualified to answer.

But seriously: clearly we can find spurious geometric patterns. Should this change our understanding of representation geometry? I’d note a few caveats first:

While the Gram matrix of my spurious-circle-set is indeed beautifully circulant, the amplitude of the (sinusoidal) off-diagonals is less than with the months. I couldn’t get em up to match the months’ Gram matrix, even to within a factor of two.
This works damn well with a set of ten, but I doubt it’d work with a set of, say, 50 (though admittedly I didn’t try very hard), so Dhruva’s other geometric findings (about e.g. all the years from 1700-2020) couldn’t be spoofed in this way.

Nonetheless, it does show that doing a kind of pursuit-matching-style search for a certain low-dim PCA’d geometry will trick you unless you’ve got enough statistical constraints on your target that it won’t happen by random chance! This does rule out certain automatic-feature-finding algorithms, which has implications for research agendas like scalable interpretability.

If this description seems wordy, it’s because my job is confusing. ↩
Note: I’ve reversed the order of the rows for storytelling ease, not that it matters. ↩

Explainability as a form of system closure

Mon, 29 Jun 2026 07:00:00 +0000

There’s a question that’s ringing out all over the machine learning world these days: what the hell counts as an “explanation” of a complex system? If you listen for a while to conversations in the labs of practitioners, the forums of mechanistic interpretability, the ivory halls of theory, or even the labs of AI-savvy neuroscientists or biologists, you’ll start hearing this question thrumming behind lots and lots of more tactical conversations. Everybody wants to explain something about AI — what it’s doing, why it’s doing it, or how to make it do something different, depending on the person — and, invariably, everyone’s struggling to do it. It’s enough to make someone who’s been at it for a few years start to seriously question the premise.

I guess this difficulty isn’t all that surprising. After all, the whole conceit of neural networks was that they’re a learning system that doesn’t need human understanding of the problem structure as an input. I suppose this all really speaks to some deep human desire — some need — to understand: it was hard for us to accept that creating AI didn’t require our understanding, and now we want to go understand everything in post-production. I wonder if it’s futile.

I think that more scientists should at least consider the possibility. It’s a gloomy thought, but sometimes you need to look directly at the void. Otherwise you might walk into it.

Does there have to be an explanation for the behavior of a complex system? This is as good a first question as any. There’s usually a tacit belief in these communities that, yeah, of course, there ought to be. Let’s play this from the other side: is there a counterexample of a system that admits no human-understandable explanation?

A dynamical systems view

A nice setting to ask these questions is that of a dynamical system. Let’s suppose you have a state variable $\mathbf{x} \in \mathbb{R}^d$ evolving according to some map $\mathbf{x}_t = f(\mathbf{x}_{t-1})$. Let’s say the dynamics are deterministic to steelman our intended position. This is general enough to include every deterministic system, including the forward pass of a neural net.¹

Suppose that we begin at a state $\mathbf{x}_0$ and later end up at a state $\mathbf{x}_T$. Suppose we seek a human-interpretable “explanation” of “why” the system wound up at position $\mathbf{x}_T$ instead of some other position. Judging by what people seem to want in applied settings, a satisfying “explanation” would need to achieve the following:

allow a human to “intuitively grasp” the path followed from $\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_T$, such that the human can ascertain why the final state was roughly $\mathbf{x}_T$ without computational assistance; and
this explanation should be sufficiently complete and actionable to allow a human-interpretable recipe for intervention, again without computational assistance: for example, which direction we should give a (microscopic) perturbation to $\mathbf{x}_0$ in order to push $\mathbf{x}_T$ in a desired direction?

Forbidding computational assistance to our human here might seem strict, but this is truly the standard most of the groups named above seek (whether or not they’re explicit about this). You can use computational aids as much as you want to find the explanation, but ultimately the explanation has to live entirely in neuro — that is, in the mind of the scientist. When asked what they’re looking for, these folks (myself included) often say “I’ll know it when I see it,” which really means they want to be able to model the whole system in the mind, the way every major scientific theory works, and the way a mechanic operates on a car.

So: let’s return to the question at hand. Does there exist a satisfying “human-interpretable” explanation for any dynamical system?

An instructive case to consider is one in which the dynamics are chaotic: for example, if our system is the logistic map or a double pendulum. In a chaotic system, a tiny perturbation to the “input” $\mathbf{x}_0$ will totally change the output $\mathbf{x}_T$, and there’s no hope of giving either:

a human-interpretable account of “why” we got roughly $\mathbf{x}_T$ specifically, or
a human-interpretable recipe for intervention, i.e. which direction we should give a (microscopic) perturbation to $\mathbf{x}_0$ in order to push $\mathbf{x}_T$ in a desired direction.

As the chaotic dynamics proceed, it will become steadily harder for a human to hold any representation of the forward map $\mathbf{x}_0 \mapsto \mathbf{x}_T$ in their mind, and in neuro answers to the above will become impossible. Here’s an illustration of this principle using the Lorenz attractor, a chaotic set of differential equations in three dimensions:

$t = $ 0

$\dot{x} = \sigma(y - x)$
$\dot{y} = x(\rho - z) - y$
$\dot{z} = xy - \beta z$

$\sigma = 10$
$\rho = 28$
$\beta = \tfrac{8}{3}$

timestep t

The key point is that the complexity of the forward map increases extensively with time $T$, so no matter how much effort the human is willing to put into constructing a mental model of the forward map, it’ll become impossible after long enough (which might not actually be that long).²

A sufficiently complex forward map is not human-interpretable. In a chaotic system, the forward map steadily increases in complexity.

In summary, any chaotic system gives us a counterexample that proves that not every system even admits a human-interpretable “explanation.” This should give us pause in trying to “explain” deep neural nets: maybe it’s impossible. The universe doesn’t run on explanations, it runs on physics, and there’s no guarantee that human-understandable explanations exist.

What if we restrict to non-chaotic systems?

Chaotic systems are famously intractable, and so this might seem to offer a glimmer of hope. However, this hope is easily dashed. Consider instead the following system:

\[\mathbf{x}_t = \begin{cases} f(\mathbf{x}_{t-1}) & t \le T, \\ g(\mathbf{x}_{t-1}) & t > T, \end{cases}\]

where $f(\cdot)$ is a chaotic system and $g(\cdot)$ is non-chaotic and as simple as you like. With any finite crossover time $T$, the dynamics aren’t technically chaotic, since chaos is asymptotic. As $T$ grows, however, the forward map gets more and more complicated, and we can invoke our previous argument just fine. The root of the problem in the chaotic case wasn’t the chaos, it was just that the forward map is really complicated. Eventually it gets so complicated that a human can’t approximate it all in neuro.

The nature of explanations

The real fruit to pluck from this chaos counterexample is an intuition about what we mean by an “explanation.” The crazy thing about chaos is that there’s a sort of “upward causal flow” from microvariables to macrovariables: the precise position within a small volume of state space eventually flows upwards to greatly affect the coarse position across a large volume. (Or, if you prefer CS language: the less-significant bits in a binary representation of $\mathbf{x}_t$ eventually affect the most-significant bits.)

This means that the system isn’t closed in a finite number of variables. In a non-chaotic system, if you want to predict the leading $k$ bits of $\mathbf{x}_T$, you can do that from $O(k)$ leading bits of $\mathbf{x}_0$, regardless of $T$. In a chaotic system, you generally need $O(k \cdot T)$. This means that the leading $k$ bits don’t form a closed system, even approximately.

The big idea here is that an explanation is precisely a closed mental model of a system, or at least part of it. Since you want to hold the entire thing in your head, it’s got to take human-interpretable inputs and predict human-interpretable outputs, and thereby comprise an (approximately) closed system on human-interpretable variables. Here are some examples:

A mechanic working on a car has thousands of “car variables” in mind: different components, their shapes and materials, their ages and wear, their temperatures and affordances and interactions and potential problems. The more experienced the mechanic, the more variables they can track, but it’s always finite. (For example, I doubt any mechanic has a mental model with more than a million components. Don’t ask me how to count.) They can then mentally model a car as a closed system in these variables, and thereby make inferences based on observations as to what’s going on inside this system.
A scientific theory gives a closed mental model within human-understandable variables. For example, Newton’s laws describe the movements of objects as a closed system. Mendelian genetics describes a closed system within genes and phenotypes. It’s often said that a scientific theory has to be predictive, and this is just another side of the coin of being closed in the dynamical systems sense.
When you start to feel you “understand” a person, this is usually because you start to get a mental model of how different things around them affect their internal experience, and how that leads to different output. This is precisely an approximate model of some parts of that person as a closed system.

Of course, except in the very cleanest cases, these systems aren’t strictly closed. Newtonian mechanics isn’t closed (c.f. quantum mech and relativity), genetics (and any biological theory) is full of complexities and caveats, and heaven knows you can’t understand any part of a person as a closed system. Even the most skilled mechanic is going to be guessing a lot of the time. But approximately, with a much greater-than-chance fidelity, you can understand these things as leaky closed systems in human-interpretable variables.

Another important note is that these variables need not be obvious a priori. “Most of the work” in coming up with most scientific theories is in identifying the right variables. The requirement is that they’re human-understandable a posteriori.

In a dynamical system, the human-interpretable variables are things like “the system was in this rough region of state-space” or “the third bit of a binary representation of the state variable was a $0$” or “the system moved between regions $A$ and $B$ three times in the last $n$ steps.” Our chaotic system is inherently unexplainable because it’s (provably, mathematically) not closed in any finite set of variables.³

So can we “explain” deep learning?

None of this means that science of a complex system like deep learning is impossible. It just means that a science of a complex enough system is impossible. We should attack the problem from both sides, identifying the simple parts we can understand.

What we should be looking for is precisely approximate closures in human-understandable variables. The challenge of the scientist looking at a complicated system is precisely to identify a set of human-graspable variables and their interactions. We should remain humble in that regard and be clear that we seek avenues towards finding those variables and describing the system they comprise.

For example, at its best, mechanistic interpretability finds human-understandable interactions between intuitive variables in model input, weights, internal states, and outputs. (Think: “input text of one of a few types goes in; model weights and internals comprise certain circuits; model output is predictable by a human given an understanding of these circuits.”) At its worst (and perhaps its median), mechinterp is confused as to what variables it describes, how they interact in a closed system, and what the path to a full understanding is.

At its best, fundamental deep learning science finds closed interrelationships between intuitive mathematical quantities, usually system hyperparameters or macroscopic statistics. (Think: “Hessian sharpness steadily increases to $2 / \eta$ before it is stabilized there by the third-order curvature of the loss landscape” or “to preserve key macroscopic stats, layerwise learning rates should scale a particular way with layer widths.”) At its worst (and, yes, perhaps its median), it is confused about its variables of study, their potential interrelationships, and its goals. We tried to lay out a vision of the best fundamental science of deep learning in our recent perspective paper, “There Will Be a Scientific Theory of Deep Learning,” and a huge through-line in that paper is the identification of important macroscopic stats and the description of their interrelationships.

The reality is that, when facing a complex system, it’s probably going to be the case that some part of it is human-understandable, and some part of it is going to be too complex. (This is all but inevitable, since system complexity is in principle unbounded, but a human’s capacity to hold a detailed mental model in neuro is finite.) Of course, the more effort the human’s willing to put into developing their understanding, the more variables they might hold in their head, and the more of the system might become understandable, provided those variables dynamically close. Some systems are easier to understand than others, and many (most?) complex systems admit both an easy shallow understanding and more challenging deeper understandings.

We should just be very clear-eyed going in that we’re not trying to understand everything, it’s possible that we can’t construct an in neuro explanation for some things, and that we’d better be clear what we’re going for, lest we waste a lot of time trying to do the impossible. Seems to me that where a lot of time gets wasted is when people see a human-interpretable output (for example, “this self-driving car ran a red light”) and expect that this must admit a human-interpretable explanation. Well, the example of the Lorenz attractor proves by example that the universe doesn’t owe you any such explanation: even though “output” observations like “$\mathbf{x}_T$ falls in the left lobe of the butterfly attractor” are human-interpretable, they’re causally downstream of a huge number of variables that aren’t, and thus aren’t part of a closed system of a finite number of human-interpretable variables, and thus don’t admit predictive explanations of the sort we want in most scientific applications.

This essay was inspired by a visit to CHAI at which Ram Rachum presented work on explanations for RL systems. The idea of system closure came out of a conversation with Julian Yocum, who (if I recall correctly) offered it in rebuttal when I invoked chaos to argue that there need not exist satisfying explanations.

And the full training process, given the relevant random seeds. ↩
Don’t get me wrong, there’s a lot we can understand about the Lorenz system: the shape of the attractor, the fact that it is chaotic, the Lyapunov exponents, and more. I don’t mean to say that we simply cannot “understand” the Lorenz system in any way. But the nature of the chaotic system means that we really can’t “understand” any of the dynamical variables in the human-interpretable, mental-model, fully-in-neuro way that many communities hope to understand deep learning. (Note that since every trajectory ends up near the attractor, being on the attractor isn’t a dynamical variable, it’s a constant.) We only feel we understand chaotic systems like these because we know that certain types of understanding or prediction are impossible, so we know not to try. The question this should make you ask about the complex system you’re studying in your research is: which types of human-interpretable explanation are actually going to be possible? ↩
You might feel you “understand” the Lorenz attractor in the graphic above. For example, you might point out that after a sufficiently long time, the system will be found near the famous butterfly-shaped strange attractor, and that mathematical properties of that attractor may be characterized. This is fair. For the purposes of this essay, however, this really just means that some leading-order bits of the system are fixed — at large $T$, they’re no longer even dynamical variables — and all the true dynamical variables don’t form a closed system. We only feel we understand the dynamics because we can satisfy ourselves that we could never predict the real dynamical variables. Let’s not kid ourselves. ↩

Higher derivatives of an inverse function

Tue, 16 Jun 2026 07:00:00 +0000

I learned something beautiful today and just wanted to share it. Obviously, this blog is super eclectic, with tons of posts on science and math and art and life, with no particular order. Maybe someday I’ll organize it.

This a pretty basic calculus trick. In fact, I wouldn’t be surprised if I’d come across it in high school and just forgot it. But today it came up in my research, and I had to rederive it, and I thought it was very pretty.

So say you have a scalar function $s(t)$, which we’ll assume to be differentiable and generally cooperative. You can go ahead and find its derivatives $\frac{d s}{d t}, \frac{d^2 s}{d t^2}, \frac{d^3 s}{d t^3}$, and so on. But suppose that (like I did) for some problem at hand, you need the derivatives in the other direction: $\frac{d t}{d s}, \frac{d^2 t}{d s^2}, \frac{d^3 t}{d s^3}$ and so on.

(It’s worth noting that the inverse function $t(s)$ doesn’t actually have to exist globally for this to work — that is, $s(t)$ need not be one-to-one. It just needs to be one-to-one locally, so you can define $t(s)$ in a local region, and take derivatives, after which you can forget about $t(s)$. This amounts to requiring that $\frac{d s}{d t} \neq 0$ at the point of study, and indeed our final formulae will diverge if $\frac{d s}{d t} = 0$, indicating this requirement.)

The first derivative. Delightfully, it holds that

$$ \frac{ d t}{ds} = \left( \frac{ds}{dt} \right)^{-1}, $$

as you can easily confirm by drawing a scalar function and transposing the two axes. I find this result really great because you can just treat $\frac{ds}{dt}$ as a conventional fraction and take its inverse. Shows it’s great notation.

The second derivative. This one we need to be more careful about. Our main tool is the chain rule: $\frac{d}{ds} = \frac{dt}{ds} \cdot \frac{d}{dt}$. Applying this, we find that $\frac{d^2 t}{ds^2} = \frac{d}{ds} \frac{d t}{d s} = \frac{d t}{d s} \cdot \frac{d}{dt} \frac{d t}{d s} = \left( \frac{d s}{d t} \right)^{-1} \cdot \frac{d}{d t} \left( \frac{d s}{d t} \right)^{-1}$, and evaluating the final derivative we find that

$$ \frac{d^2 t}{ds^2} = - \left( \frac{d s}{d t} \right)^{-3} \frac{d^2 s}{d t^2}. $$

I find this really nice — the second derivatives are proportional, with the rather surprising ratio of $- \left( \frac{d s}{d t} \right)^{-3}$. Even though it didn’t solve the particular research problem I hoped it would, seems like it might eventually be useful for something. (Wonder if there’s some intuition for that other than a dimensionality argument…)

The third derivative. Again applying the chain rule, taking $\frac{d^3 t}{ds^3} = \frac{dt}{ds} \cdot \frac{d}{dt} \frac{d^2 t}{ds^2}$, we find that

$$ \frac{d^3 t}{ds^3} = \left( \frac{d s}{d t} \right)^{-5} \left[3\left(\frac{d^2 s}{d t^2}\right)^2 - \frac{d s}{d t} \cdot \frac{d^3 s}{dt^3}\right]. $$

Not as pretty, and you lose the nice proportionality between $s^{(k)}(t)$ and $t^{(k)}(s)$. Thanks to the chain rule, things only get more complicated from here on out.

The polynomial features of random neural networks

Wed, 18 Mar 2026 07:00:00 +0000

The language of polynomials is beautiful and well-developed: millenia of work have furnished a rich, deep theory that spans analysis and algebra, with basics we teach to children and high reaches still under development. The language of neural network features has not yet been written, but it promises to be beautiful: empirics show that hidden features can represent complex concepts and form striking circuitry. Fortunately, more and more results in fundamental deep learning theory suggest that the language of polynomials might actually be the language of hidden features, at least to a substantial fraction. How wonderful if we could fathom the depths of learning and cognition with polynomial algebra!

My (biased) favorite example of this confluence of languages comes from my recent paper, which will appear at ICLR in April. We performed a first-principles study of the inductive biases of simple learning methods and found that (a) they treat real image datasets as Gaussian blobs, and that (b) learning on these blobs decomposes into independent learning of Hermite polynomial coefficients. What falls out is a surprisingly simple and beautiful theory of inductive bias which makes excellent (i.e. correct) predictions of test-time performance on different tasks. Just look at these curves which we predict using our polynomial language:

A figure from our paper Karkada*, Turnbull* et al. (2026) showing extremely good prediction of kernel regression learning curves from minimal dataset statistics.

You might be thinking: reading papers is hard, and that one looks quite technical. You might then be wondering: is there a simpler way to see how polynomial language arises in this context? If that’s so, dear reader, then you have clicked on the right blogpost. The purpose of this note is to walk through the deep underlying reason why Hermite polynomials arise in the description of neural network hidden features.

Setup

Consider a data vector $\mathbf{x} \in \mathbb{R}^d$ and a nonlinear transformation $\mathbf{h}(\mathbf{x}) := \sigma(\mathbf{W}\mathbf{x})$ which embeds it in a feature space. Here $\mathbf{W} \in \mathbb{R}^{n \times d}$ is a linear transformation and $\sigma(\cdot)$ is an elementwise nonlinearity. For the picture we want to develop, it’s enough to let $\mathbf{W}$ be random with $W_{ij} \sim \mathcal{N}(0, \frac{\alpha}{d})$. No polynomials yet: since we didn’t require $\sigma$ to be a polynomial, each hidden activation $h_i(\mathbf{x})$ is a complicated, nonpolynomial function.

Dot-product kernels have polynomial features

The key idea is that, given one assumption, these secretly are polynomial features up to rotation. After a suitable orthogonal transformation on $\mathbf{h}(\mathbf{x})$, we get polynomial features. This is most easily seen with kernel language: take a second data vector $\mathbf{x}’$ and consider the feature kernel $K(\mathbf{x}, \mathbf{x}’) := \frac{1}{n} \langle \mathbf{h}(\mathbf{x}), \mathbf{h}(\mathbf{x}’) \rangle$. Let us make the following assumption on this kernel:

Assumption A (dot-product feature kernel). For all data vectors $\mathbf{x}, \mathbf{x}'$, the feature kernel admits the following form: $$ K(\mathbf{x},\mathbf{x}') = K(\langle \mathbf{x}, \mathbf{x}' \rangle) = \sum_{\ell \ge 0} \frac{c_\ell}{\ell!} \langle \mathbf{x}, \mathbf{x}' \rangle^\ell $$ for some constants $c_\ell \ge 0$.

From Assumption A, it follows immediately that the hidden features are equivalent up to rotation to polynomial features:

$$ \exists \ \text{rotation} \ \mathbf{U} \ \text{s.t.} \ \mathbf{U} \mathbf{h}(\mathbf{x}) = \operatorname{vec}\!\left( \begin{bmatrix} \sqrt{c_0} & \sqrt{c_1}\,\mathbf{x} & \frac{\sqrt{c_2}}{\sqrt{2!}}\,\mathbf{x}\mathbf{x}^{\top} & \frac{\sqrt{c_3}}{\sqrt{3!}}\,\mathbf{x}^{\otimes 3} & \cdots \end{bmatrix} \right) =: \boldsymbol{\psi}_\mathbf{c}(\mathbf{x}), $$

where we have defined the prismatic operation $\boldsymbol{\psi}_\mathbf{c}(\mathbf{x})$ containing all monomial features of $\mathbf{x}$ (more on that to come in a future paper). When the data is Gaussian and the level coefficients $c_\ell$ decay quickly, Gram-Schmidt orthogonalization of the elements of this prismatic feature vector gives the Hermite polynomials (!!) as kernel eigenfunctions. It turns out real image data is often Gaussian enough for the Hermite polynomials to be a good approximation to the kernel eigensystem. This is pretty much the whole story of our paper.

When can we take Assumption A?

The problem with Assumption A is that, strictly speaking, it’s false. Random feature kernels do not take this exact dot-product form, even in the infinite-width limit. Fortunately, we can often make that assumption, and it’s not too wrong.

Let’s return to the random feature kernel above. As we take width $n \rightarrow \infty$ and take the average over hidden neurons and their associated Gaussian weights, we get the kernel

$$ K_{\sigma,\alpha}(\mathbf{x},\mathbf{x}'):=\mathbb{E}_{(z_1,z_2)\sim\mathcal{N}(\mathbf{0},\boldsymbol{\Sigma})}\!\left[\sigma(z_1)\sigma(z_2)\right],\qquad\text{where }\boldsymbol{\Sigma}=\frac{\alpha}{d}\begin{bmatrix}\|\mathbf{x}\|^2 & \langle \mathbf{x}, \mathbf{x}' \rangle \\\langle \mathbf{x}, \mathbf{x}' \rangle & \|\mathbf{x}'\|^2\end{bmatrix}. $$

This is precisely the neural network Gaussian process (NNGP) kernel of Lee et al. (2017). Remember that we’re initializing our random weights as $W_{ij} \sim \mathcal{N}(0, \frac{\alpha}{d})$, so that’s where the $\frac{\alpha}{d}$ comes in.

The NNGP kernel $K_{\sigma,\alpha}(\mathbf{x},\mathbf{x}’)$ is rotation-invariant: that is, it admits the form

\[K_{\sigma,\alpha}(\mathbf{x},\mathbf{x}') = K_{\sigma,\alpha}(\|\mathbf{x}\|^2, \|\mathbf{x}'\|^2, \langle\mathbf{x},\mathbf{x}'\rangle).\]

However, it is not a true dot product kernel; those first two norm arguments are real.¹ That said, we can ignore the first two arguments if we know they do not vary over our dataset. For example, if we know that every data vector we will care about lies on a sphere: $\lVert\mathbf{x}_i\rVert = r \ \forall i$, then we can neglect the first two arguments, and we are left with a dot-product kernel.

Of course, it’s a rare case when our data’s exactly on a sphere, but it’s often the case that the data roughly concentrates in norm, as is the expectation in high dimension. In such cases, we approximately get a dot-product kernel, and the eigensystem is approximately built from Hermite polynomials. This approximation is often very reliable in practice: in our paper, we show that assuming concentration of norm allows us to predict the eigensystem of CIFAR-10 with great accuracy (see learning curves above).

To show how robust this assumption is, it’s worth noting that the data norms of CIFAR-10 really don’t concentrate very tightly; see the figure below. This poor concentration is still good enough to do theory that assumes the norm concentrates. At least in many cases, this a robust assumption.

The data vector norms for CIFAR-10 don't concentrate tightly around their mean (dashed line). Nonetheless, eigensystem calculations based on an assumption of concentration of norm work remarkably well.

If I’m honest, I don’t really know why we didn’t have to take corrections involving the width of this distribution into account. It’s probably something that some future theory paper will have to go back and understand. Moving forward, though, we can reliably take Assumption A for real image datasets, which gives us a dot-product kernel, polynomial features, and a Hermite eigensystem.

The remainder of this post consists of some technical notes on how to calculate kernel level coefficients. They probably won’t interest you unless (and until) you need them.²

Activation function $\rightarrow$ kernel coefficients on the sphere

Let’s assume our data lies on a sphere: $\lVert\mathbf{x}_i\rVert = r$. Our NNGP kernel is then

\[K_{\sigma,\alpha}(\mathbf{x},\mathbf{x}') = K_{\sigma,\alpha,r}(\langle \mathbf{x}, \mathbf{x}' \rangle) = \sum_{\ell \ge 0} \frac{c_\ell}{\ell!} \langle \mathbf{x}, \mathbf{x}' \rangle^\ell.\]

How do the level coefficients $c_\ell$ depend on $(\sigma, \alpha, r)$? There is a simple answer: computing the Gaussian expectation above, we find that

\[c_\ell = \frac{\ell!}{r^{2\ell}} \left( \mathbb{E}_{z \sim \mathcal{N}(0,1)} \! \left[ h_\ell(z) \ \sigma\!\left( r\sqrt{\tfrac{\alpha}{d}}\, \cdot z \right) \right] \right)^2,\]

where $h_\ell(z) = \frac{1}{\sqrt{\ell!}} \text{He}_\ell(z)$ is the $\ell$-th normalized probabilist’s Hermite polynomial. In words, we’re extracting the $\ell$-th Hermite coefficient of the function $z \mapsto \sigma\left( r\sqrt{\tfrac{\alpha}{d}}\, \cdot z \right)$. That prefactor of $r\sqrt{\tfrac{\alpha}{d}}$ makes a lot of sense: it’s the standard deviation of the (Gaussian!) preactivations fed into the activation function $\sigma(\cdot)$. Hermite decompositions of a rescaled activation function often show up in this corner of theoryland.³

This is a powerful result: if you know the Hermite decomposition of $\sigma(\cdot)$, then you know the level coefficients of your kernel and the relative weights of the different orders of monomial features. The Hermite decompositions of common activation functions (e.g. $\text{ReLU}$) can be worked out in closed form. Moreover, a lot can be learned from symmetries and taking note of which Hermite coefficients are exactly zero. For example, $\sigma = \tanh$ is odd, so it will lack all even coefficients: $c_{2\ell} = 0$. In a weirder case, $\sigma = \text{ReLU}$ equals a linear function plus an even function, so it has $c_\ell = 0$ for $\ell = 3, 5, 7, \cdots.$ This particular pathology of $\text{ReLU}$ has cropped up many times in my research.

Activation function $\rightarrow$ kernel coefficients near the origin

Now things are going to get somewhat subtle. So far, when we’ve talked about concentration of norm, we’ve really meant that the norm of the data vectors concentrates in a relative sense, not in an absolute sense. That is, we care that

\[\frac{\max_i \|\mathbf{x}_i\|}{\min_j \|\mathbf{x}_j\|} \approx 1, \qquad \text{not that} \qquad \max_i \|\mathbf{x}_i\| - \min_j \|\mathbf{x}_j\| \approx 0.\]

However, there is a very important solvable limit in which the data vectors concentrate in norm absolutely but not relatively. This is the limit in which the data vectors all have small norm: $\lVert\mathbf{x}_i\rVert \ll 1$. If all the vectors have the same small norm (e.g., $\lVert\mathbf{x}_i\rVert = \lVert\mathbf{x}_j\rVert = r \ll 1$), or the variation is also small in a relative sense, then we recover the case above. However, what we really want is an analysis valid for all $\mathbf{x} \in r B^{d}$, where $r \ll 1$ and $B^d$ is the $d$-ball.

The method of attack here is direct: simply examine the kernel $K_{\sigma,\alpha}(\lVert\mathbf{x}\rVert^2, \lVert\mathbf{x}’\rVert^2, \langle\mathbf{x},\mathbf{x}’\rangle)$ and naively (Taylor-)expand it for small $\lVert\mathbf{x}\rVert^2, \lVert\mathbf{x}’\rVert^2$. Upon trying this, one quickly makes a few observations:

It is not always possible to take this expansion analytically. For example, the Laplace kernel $e^{-\lVert\mathbf{x} - \mathbf{x}’\rVert}$ is not smooth around zero in these arguments.
This smoothness is related to the smoothness of the activation function $\sigma$ around zero: if the activation function is analytic around zero, so is the kernel.
The derivatives of $\sigma$ determine the coefficients of the kernel’s Taylor series.

In particular, one finds that so long as the activation function is analytic (on all of $\mathbb{R}$) and integrates nicely against Gaussians, the NNGP kernel is given exactly by

$$ \begin{aligned} K_{\sigma,\alpha}(\mathbf{x},\mathbf{x}') &= \sum_{\ell\ge 0} \frac{1}{\ell!} \left(\frac{\alpha}{d}\right)^\ell \langle \mathbf{x}, \mathbf{x}' \rangle^\ell \left( \sum_{q\ge 0} \frac{\sigma^{(\ell+2q)}(0)}{2^q q!} \left(\frac{\alpha}{d}\right)^q \|\mathbf{x}\|^{2q} \right) \left( \sum_{q'\ge 0} \frac{\sigma^{(\ell+2q')}(0)}{2^{q'} q'!} \left(\frac{\alpha}{d}\right)^{q'} \|\mathbf{x}'\|^{2q'} \right). \end{aligned} $$

In the above, $\sigma^{(k)}(0)$ denotes the $k$-th derivative of $\sigma$ evaluated at zero. When the norms are small ($\lVert\mathbf{x}_i\rVert, \lVert\mathbf{x}_j\rVert \ll 1$) the $q, q’ = 0$ terms dominate those with $q, q’ \ge 1$, and we can heuristically approximate

\[K_{\sigma,\alpha}(\mathbf{x},\mathbf{x}') \approx \sum_{\ell \ge 0} \frac{c_\ell}{\ell!} \langle \mathbf{x}, \mathbf{x}' \rangle^\ell \qquad \text{where} \qquad c_\ell = \left(\frac{\alpha}{d} \right)^\ell \cdot \left( \sigma^{(\ell)}(0) \right)^2.\]

This is a main result in what I think of as “wide kernel theory”: the study of kernels which are much wider than the data distributions on which they’re evaluated, often leading to great simplifications and nice polynomial forms.⁴

Some examples for different choices of $\sigma$

Here we report values of these small-norm level coefficients for different activation functions. Note that care must be taken to correctly incorporate the right power of $\alpha/d$.

$$ \begin{array}{c|ccccc} & c_0 & c_1 \cdot \left(\frac{\alpha}{d}\right)^{-1} & c_2 \cdot \left(\frac{\alpha}{d}\right)^{-2} & c_3 \cdot \left(\frac{\alpha}{d}\right)^{-3} & c_4 \cdot \left(\frac{\alpha}{d}\right)^{-4} \\[0.6em] \sigma(z) & \sigma(0)^2 & \sigma'(0)^2 & \sigma''(0)^2 & \sigma^{(3)}(0)^2 & \sigma^{(4)}(0)^2 \\ \hline \sqrt{2}\cos\!\left(z+\dfrac{\pi}{4}\right) & 1 & 1 & 1 & 1 & 1 \\ e^z & 1 & 1 & 1 & 1 & 1 \\ \tanh(z) & 0 & 1 & 0 & 4 & 0 \\ \mathrm{sigmoid}(z) & \dfrac{1}{4} & \dfrac{1}{16} & 0 & \dfrac{1}{256} & 0 \\ \tanh\!\left(z+\dfrac{1}{2}\right) & 0.214 & 0.619 & 0.528 & 0.319 & 15.620 \\ \mathrm{ReLU}(z) & \mathrm{NA} & \mathrm{NA} & \mathrm{NA} & \mathrm{NA} & \mathrm{NA} \end{array} $$

Derivatives of different activation functions.

I used GPT to help with some of the math here. I’ve sense-checked all of it, and the final results have already fed into successful experiments I’ve run, so I feel pretty confident in this, but I own any errors.

It can be proven that only in the trivial affine case $\sigma(z) = a z + b$ is the NNGP kernel exactly a dot-product kernel. ↩
Looking at you, Joey Turnbull. ↩
This connection between the Hermite decomposition of the activation function and the level coefficients of the resulting NNGP kernel was first pointed out by Daniely et al. (2016), and the NTK version was later given by me and friends in Simon et al. (2021). I no longer believe in the importance of that paper’s central story, but our Theorem 3.1 is still great. ↩
Worth noting that, while I think this heuristic approximation will be pretty much fine, there are some subtleties involved; in particular, since kernels are inverted when they’re used, dropping a small term can change that inverse unless it falls “in the shadow” of larger terms. Dropping these terms might change learning on a small number of norm-sensitive target functions. ↩

Goo

Fri, 06 Feb 2026 08:00:00 +0000

I love classic folk and country songs. I listened to so many at camp growing up, songs like “Rocky Top” or “Country Roads” or John Prine’s “Paradise” that tug at the heartstrings and make you yearn for the simple things. A really good folk song gives you the feeling that it’s been around forever. It’s like a pebble in a creek, worn smooth from the tumbling of being sung again and again.

In 2018, I wanted to see if I could write one. I thought, well, somebody wrote these songs, and trying to fake the timeless feel sounded fun. My songwriting process started in Albania, of all places, when I was finally bored enough for inspiration to strike. A few weeks later I had a finished song. It was my first song with lyrics. I’ll show it to you at the end.

I started out writing a slow, soulful ballad about a mother’s love for a child about to venture out on his own. I tried to establish a sense of place (“This river, my child, is winding and wild…”), and make plain the mother’s fondness for her son, and her regret but acceptance of his leaving her because that’s the way of the world. I wrote a verse like that. Then I got stuck.

Truth is, I found it kind of painful to be that sincere. I wasn’t really able to show my feelings back then, and it felt so, so uncool to be that simple and direct without twisting into irony or flexing my intellect. The result was that I wrote one verse of a sincere ballad about a mother’s love, and then two verses and three choruses where the mother goes on to tell the child that the one thing they should really remember is that, if they ever get in trouble, they should eat a big blob of whatever goo they find lying around.

I told myself I was making fun of folk superstition and old wives’ tales, of motherly chidings to not walk past black cats or under ladders. I don’t think that really shines through. It’s a fun song and one I still listen to fondly, but I cringe a bit. I wasn’t much of a storyteller yet, and there are some things I know now that I didn’t back then. I wanted to write a song poking fun at how true motherly love is blended together with the most ridiculous advice (@Mom). Instead of switching from real wisdom all the way over to goo-mode, I’d interleave the two: a few lines of good advice, then wait, hold the phone, yes, it’s back to goo. It takes genuine love to hold both things and balance them in writing.

Well, you can’t change the past. As promised, all the way from 2018, here’s Goo.

Your browser does not support the audio element.

Credit to my brother Sam for the guitar backing. The other instruments are actually the same harmonica, played way low for the bass and high for the solo. Thanks also to Bennett Witcher, who’s sent this recording back to me multiple times after I’ve lost it. For your services, I hereby confer upon you the title of Eternal Steward of the Goo.

Predicting kernel spectra powerlaw exponents from raw data statistics

Thu, 29 Jan 2026 07:00:00 +0000

This is a technical note on kernel theory. Venture in at your own risk!

My collaborators and I recently put out a paper called Predicting kernel regression learning curves from raw data statistics in which, among other things, we show how to map from the data PCA stats to the eigenfunctions and eigenvalues of a broad swath of rotation-invariant kernels. It’s a surprising result, both mathematically and empirically: the formulas we find are very simple, and they predict very well for both synthetic Gaussian datasets and image datasets like CIFAR-10. This has given me real hope that we can deal with the problem of data in the fundamental science of deep learning.

At about the same time, Arie Wortsman and Bruno Loureiro published a paper that takes up a very similar topic! They also ask about the mapping from data covariance stats to kernel eigenvalues, and where we discuss the same questions, we find agreeing answers. They’re particularly interested in powerlaws (while we’re more spectrum-agnostic), and they’d like to know: if the data covariance eigenvalues decay as a powerlaw with exponent $\alpha$, with what exponent do the kernel eigenvalues decay?

This is a great question, well-motivated by recent developments in practical machine learning and attempts to explain them. The data decay exponent is a more fundamental object, but the kernel eigenvalue decay exponent is the more practically important one. It’s a good question to ask how they’re linked. In this blogpost, I’ll review their answer, propose an alternative perspective with some experiments to back it up, and discuss what I see as important open questions.

Review: the answer of Wortsman and Loureiro (2025)

Let the data be sampled $\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Gamma})$ with $\boldsymbol{\Gamma} = \mathrm{diag}(\gamma_1, \ldots, \gamma_d)$. Let the data covariance eigenvalues be $\gamma_i = i^{-a}$ with exponent $a > 1$. (We could add a constant of proportionality — that is, $\gamma_i = \gamma_0 i^{-\alpha}$ — but this only changes the results trivially, so I’ll assume $\gamma_0 = 1$ to keep things simple.) The data dimension $d$ will be very large; for the purposes of this blogpost, we will let it be infinite.

Wortsman and Loureiro find that the kernel eigenvalues asymptotically decay as $\lambda_j \sim j^{-a}$ — that is, the kernel spectral decay exponent is the same as the data covariance decay exponent. They give nice formal bounds that imply this, and they also give a beautiful characterization of the spectrum when $d$ is finite and $a < 1$, which is very mathematically satisfying (if less practically relevant, since in with real data one almost always finds that $a> 1$).¹

Turning to experiment: can we see this?

I find that, when studying machine learning, it’s always a good idea to go check your results against experiment. So what happens if we try this prediction with a real kernel function? To try this, I drew $n = 10^4$ samples of Gaussian data with eigenvalues $\gamma_i \propto i^{-a}$ with exponent $a= 1.5$ and unit trace $\sum_i \gamma_i = 1$, with data dimension $d = 10^4$. I then computed the spectrum of a Gaussian kernel $K_\sigma(\mathbf{x}, \mathbf{x}’) = e^{-\frac{1}{2 \sigma^2} |\mathbf{x} - \mathbf{x}’|^2}$ with different kernel widths $\sigma$ over this dataset and plotted the spectrum. Here’s what I found:

A few observations. First, the spectrum is… kind of a nice powerlaw in the middle? But the transient effects at the head and finite-size effects at the tail make it annoying to tell; we’ll have to deal with these. Second, even if you squint, it’s hard to argue that the decays all look like powerlaws with $a = 1.5$. Third, the decay rates actually seem to change: the plots with larger $\sigma$ seem to have a faster decay and a larger exponent. This wasn’t in the prediction of Wortsman and Loureiro (2025). What gives, and can we predict the decay rates we actually see here?

The HEA: an alternative approach

I’m going to argue that, yeah, their answer that “the spectral decay rate is always $a$” is probably right asymptotically as you look very deep into the spectrum, but you’ll often have to look prohobitively far into the spectrum before you approach that decay rate. To do so, I’ll introduce a mathematical tool for getting the precise shape of the decay curve, even at small values of the index $j$.

The main technology we’ll use to do this is the Hermite eigenstructure ansatz (HEA) from our paper. I won’t go through it in detail here; see the paper or Joey’s blogpost. The main point we’ll use here is that the HEA gives nice, explicit formulae for the kernel eigenvalues in terms of some kernel-derived coefficients (which are basically Taylor series coefficients). For the Gaussian kernel with width $\sigma$, we have $c_\ell = e^{-\frac{1}{\sigma^2}} \cdot \sigma^{-2 \ell}$. Then

\[\left\{ \lambda_j \quad \text{for all } \quad j \in \mathbb{N} \right\} = \left\{ c_{|\boldsymbol{\alpha}|} \cdot \prod_i \gamma_i^{\alpha_i} \quad \text{for all } \quad \boldsymbol{\alpha} \in \mathbb{N}_0^d \right\}. \tag{HEA}\]

In words: the set of all kernel eigenvalues (the left-hand side) equals the set enumerated on the right, where the enumeration runs over all multi-index vectors $\boldsymbol{\alpha} \in \mathbb{N}_0^d$ and each choice of $\boldsymbol{\alpha}$ gives a certain monomial of the data covariance eigenvalues $\gamma_i$. This is the deep idea from our paper, and it unifies a lot of other cases with known kernel spectra. I won’t explain why it’s true here, but there is a pretty satisfying derivation we give in our paper.²

The HEA isn’t exact, and it isn’t perfect, but in our paper, we show that it’s empirically very accurate in basically every case we hoped it would be, including Gaussian-data cases like the one we’re studying here. Let’s check: I’ll add the theoretical spectrum predicted by simple enumeration of the HEA eigenvalues to the plots.

Pretty good! It’s not perfect, but it’s pretty close, and the discrepancy in the tails is probably due to finite-size effects in the experiment. I trust the HEA more there, and the HEA seems to show a fairly nice powerlaw that spans several decades! It looks reasonably like the eigenvalues actually follow a powerlaw with a different slope $a’ \neq a$. I didn’t draw those lines because I wasn’t totally sold, but I’m close to buying it.

To proceed, let’s just assume the HEA gives us the exact spectrum and ask what that implies.

From the HEA to spectral decay?

Now that we just have a closed-form construction for the kernel spectrum, all that remains is to look at its decay w.r.t. index. The right-hand side of the HEA is a sum over multi-indices —- it naturally takes the shape of a $d$-dimensional array —- so the challenge is just flattening this set down to a one-dimensional sorted list.

Treated as a discrete math problem, this is difficult. I’m a physicist, though, so when I see a problem like this, I’ll first try a continuum approximation to the sum and see if that works.

We want a function for $\lambda(j)$, the eigenvalue at index $j$. It turns out it’s easier to go after $j(\lambda) = \#[\textrm{eigenvalues } \ge \lambda]$ and then invert it. To get $j(\lambda)$, a decent idea is to do a Laplace’s method “saddle point” analysis: most of the total number of eigenvalues less than $\lambda$ will occur at or near a particular eigenvalue order $\ell_*$, and Taylor-expanding the multiplicity locally around $\ell_*$ turns out to be a useful idea.

It turns out that ChatGPT can just do this whole calculation if you throw it the problem setup. After some minutes of thinking, it reports that:

\[\lambda(j)\ \propto \ f(j; \sigma, a) \equiv j^{-a}\;\exp\!\Big(2a\,\sqrt{r\,\log j}\Big)\;(\log j)^{-\,\frac{3a}{4}},\]

where $r = \Big(\sigma^{-2}\gamma_1\Big)^{1/a}$. So does this work? Let’s add it to the plot:

Not really. I’m not sure why. Even after adding more terms (i.e. considering a higher-order expansion around the saddle point), it doesn’t work. It’s possible that GPT’s calculation was just wrong; I haven’t checked in detail (and that’s a first thing to do were I to take up this problem seriously). It’s better than just assuming a powerlaw with exponent $a$, though.

Let’s ignore the discrepancy and study the function $f(j; \sigma, a)$ anyways. A first observation is that it actually does asymptotically decay like $j^{-a}$. A second observation, though, is that it can take a really, really long time to get there. Here’s a plot of the slope on a log-log plot:

Slope on a log-log plot of $\lambda(j) = f(j; \sigma, a)$ with $a = 1.5$. Note that it takes an extremely long time to converge to the asymptotic slope of $-a = -1.5$!

When $\sigma$ isn’t particularly big, sometimes it takes up to $j \sim 10^{100}$ to actually get close to a decay rate of exponent $a$. Perils of asymptotic analysis! Really makes me suspect that all the power laws we see in machine learning are actually just locally powerlaws, and over enough orders of magnitude, they’ll change.

What now?

I’d say the big open question here is: given that the HEA eigenvalues look pretty good, what analysis can you do on top of them? It’s probably not the right idea to jump all the way to the asymptotic decay rate. Instead, it’ll probably turn out that the decay rate is roughly constant for the first several decades (say, from $10^1$ to $10^6$), and that’s the more interesting value to predict, since it’s the one we’ll see in experiments.

The second half of Arie and Bruno’s paper is a treatment of how the data spectrum drives generalization in KRR, which is one step beyond the kernel spectrum! I’m just not discussing it because I’m focused on the powerlaw spectrum question here. ↩
For an alternative approach, you can actually see Arie and Bruno’s Proposition 1 — they find essentially the HEA’s eigenvalue prescription in almost the same language! Their result involves constant-factor bounds on the eigenvalues (while we give the eigenvalues including constant factors), but this price is paid in exchange for their actually proving their result rigorously, where we do not except in two limiting cases. Which of these you’ll want to use probably depends on your style! If you just want a formula to try in experiments, our presentation is probably preferable. If you want to prove theorems or build rigorous mathematical foundations, our results should be a good guide for the intuition, but their analysis is probably a better place to build from. ↩

Why I am not an experimentalist; or, making faces with photolithography

Thu, 22 Jan 2026 07:00:00 +0000

In 2019, I spent six months in Gothenburg, Sweden at the Chalmers Institute of Technology. I was working in the lab of Per Delsing doing nanofabrication in my gap year before grad school, and it was not going well.

Have you ever wondered how computer chips are made? A chip is basically a block of silicon covered in an immense tangle of thin wires and pieces of doped silicon. These wires are absolutely tiny, just nanometers wide, which is way smaller than the spatial resolution of any machine we could use to mechanically lay down the wires.

To get around that and get all those wires down, you use a bewildering combination of chemistry and particle physics. First you cover the slab of silicon with a thin layer of a substance called “resist.” The resist is sensitive to either photons or electrons, so you shine either a laser or an electron beam over the slab where you want wires to go. You can then wash away the irradiated resist, exposing the bare silicon underneath, and just cover the whole thing with a thin layer of metal shot in from above. The resist blocks the metal everywhere but in the wire tracks, so when you wash away the rest of the resist, you’re left with nice, beautiful wires. Here’s a cartoon I drew:

With this technique and variations on it, all modern computer chips are manufactured. My job that fateful summer was to use this technique to design quantum inductors, and I was rapidly learning that I was not cut out for labwork. Every step in the cartoon above is actually ten different steps involving mixing toxic chemicals, heating things, putting things into big scary machines, and using inscrutable computer interfaces with a hundred settings. Oh, and to make matters worse, you have to do all this while sweating inside a head-to-toe cleanroom lab suit. My mentor, Anita, would demonstrate something with practiced care and deliberation, she’d walk away, and I would promptly forget everything except maybe what colors were supposed to be created. I cannot overstate the extent of my incompetence: on several occasions, I left a piece of support plastic called a “spider” on a hot plate and literally melted it to the piece of silicon I was working with. This continued until Anita told me to stop using spiders.

Even though I was close to useless in the lab, it turned out I was pretty good at coming up with things for other people to try. At the start of my time there, I learned that quantum inductors are usually made with three shoot-down-a-layer-of-metal steps to make three different parts of the circuit. You can do the first two in one machine cycle, but for annoying geometric reasons, everyone did the third step in its own cycle, which could take an extra day or two. I found a way to do them all in one machine cycle by shooting the metal down from three different angles. Thanks to the hard work of my collaborators to show this idea actually worked — figuring out how to fabricate the design, characterize it, and test it in quantum circuits — this is now among my most-cited publications. The finished quantum inductors looked like this:

A Josephson junction made with our speedy fab process.

And now we get to my favorite part of this story. Our silicon wafers were about two inches wide. They’re pretty expensive, so you don’t want to waste them, but you only ever end up using about two thirds of the surface area since you want to leave the borders clear so you can grab the thing with tweezers. This leaves a lot of room for you to print tests or calibration marks, and this fact was not lost on me. Do you see where I’m going with this?

It was not too long until I realized that the time it took to pattern something onto the silicon was proportional to its size, not its complexity, so I could print really complicated stuff, and whatever I wanted, onto the edges of these wafers, so long as it was small. So I figured out how scripting worked in the design software, and I wrote some Python to break images down into a format suitable for this software, and then I could print whatever I wanted in a nanometers-thick layer of aluminum. Naturally, the first thing I printed was my professor’s head:

Left: Professor Per Delsing. Right: Professor Per Delsing on silico.

I left this image up on his door. I’m pretty sure he knew it was me, but we never talked about it. I also printed my own head and a picture with some of my friends:

Top: me. Bottom: me and friends.

It’s pretty fun to be able to fabricate things at such a small scale. Each of these images has a total size of about 1mm. I still have the picture of myself, and I own a little microscope I can use to look at it. I used it on my dating profile for a while until I conceded nobody knew what the hell they were looking at and gave up.

In August 2019, I left Sweden and started my PhD at Berkeley. I switched gears away from experiment, and then away from quantum physics altogether, and joined a lab studying, among other things, the fundamental theory of deep learning, and that got me to where I am today. But now, as I use a computer, and run things with GPUs, and stand by as tech giants fill warehouses with upwards of millions of GPUs and $10^{17}$ transistors total, it’s illuminating and remarkable and a deeply privileged feeling to have gotten to see how computer chips are made.

The Ballad of the Crane Breeder

Sun, 18 Jan 2026 12:00:00 +0000

My brother’s girlfriend’s dad, Tom Zimmer, had the craziest job out of high school I’ve ever heard of. He shipped up to a wildlife preserve in Wisconsin run by the International Crane Foundation and there found his job was, among other things, to artificially inseminate cranes. They wouldn’t breed in captivity, so he had to go to their pens — the male first, then the female — to collect and deposit the necessary genetic material. One hitch: they’re such violent birds (especially the males) that they’ll peck the hell out of you before you can get the goods, so he’d cover the backs of his calves with cardboard for armor, catch them between his legs when they ran at him, and do what he came to do. This is apparently a legitimate technique in crane husbandry.

Of course, a man with a story like that is in desperate need of a songwriter, and I decided I fit the bill. Please enjoy my rendition of the teenage occupation of one Tom Zimmer.

Your browser does not support the audio element.

Credit to Isaac Harris for the guitar accompaniament, my uncle Kevin Simon for help with the “bugle” solo, and of course Tom Zimmer, to whom I have promised royalties.