Best Paper Award at the ICML 2026 Workshop on Weight-Space Symmetries
Parameter symmetries determine representational geometry in overparameterized nonlinear networks
Two networks can compute the exact same function yet have radically different representations, and two networks computing different functions can have nearly identical ones. We show why this happens, why overparameterization amplifies the issue, and how efficiency constraints can restore a unique geometry that reflects what the network actually computes.
Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such inferences presume a meaningful link between representational geometry and the computation being performed. For artificial neural networks, however, the extent to which function constrains representation remains unclear. One key obstacle is that these networks admit parameter symmetries: changes in parameterization that preserve function exactly while reshaping representational geometry. Here, we show that a broad class of parameter symmetries acts on representations through just three primitive feature transformations: addition, duplication, and scaling. This feature-level characterization yields a closed-form decomposition of representational geometry into essential and auxiliary components, which makes precise how degeneracy in representational geometry can grow with overparameterization even when function is held fixed. Finally, we show that implementation-level selection rules can resolve this degeneracy, yielding identifiable geometries in which features are weighted according to their contributions to the network's function. Together, our results delineate when representations can support inferences about computation, and when they cannot.
