Papers for

neural network architects

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Weight-space programming solves a ladder of two-player games zero-shot

Zero-Shot Weight-Space Programming Across a Difficulty-Graded Ladder of Two-Player Games

Abstract: Weight-space programming sets every weight of a feed-forward network by hand, from an explicit specification of the desired computation, with no gradient descent and no training data anywhere in the construction. We ask how far a large language model can carry this practice, zero-shot, across games of deliberately increasing difficulty. We design a four-rung ladder (Nim, tic-tac-toe, two sizes of Connect Four, and a chess endgame) spanning four axes of difficulty, and build a second, independent network at each rung, using a structurally unrelated game at matched difficulty, to separate what a rung's finding says about the difficulty level from what it says about the one game used to measure it. Ten networks are built and independently verified. Six admit a complete, non-statistical proof of correctness against an independent oracle. The chess-endgame rung is exact where correctness is cheap, move legality and, in one network, an exact one-ply mate detector, and collapses where it is expensive, composing a multi-move mating plan, from 84.1-100% success at the one-ply tactics it was built to get right down to 0.57-9.13% in full adversarial play; its replicate shows this collapse is an order of magnitude less severe under a shorter required plan and exact rather than heuristic safety checks. The standard-sized rung confirms, by holding hand-set evaluation weights fixed and adding only shallow search, that the remaining gap is an architectural ceiling rather than a matter of engineering effort, and this exact collapse reproduces across two structurally unrelated games. We release all ten networks, their independent oracles, construction-cost measurements, and a cold-prompt protocol for a future cross-model comparison

Fri 6 NovArtificial Intelligence
The gist
This paper studies whether large language models can set the exact weights of neural networks by hand to play games, without any training or trial and error. The authors build networks that play games of increasing difficulty, from simple Nim to a chess endgame, and check their moves with independent proofs or oracles. They find that the networks work perfectly at simpler levels, but performance drops at harder chess tasks, showing limits in the approach and architecture. Their work also releases all their networks and tools for future comparison.
Open → 2611.09271v1

Tight bounds reveal complexity advantage of kolmogorov arnold networks

Tight Asymptotic Bounds on the Maximum Number of Linear Regions of Piecewise-Linear Kolmogorov--Arnold networks

Abstract: Schoots et al.~\citep{schoots2025relating} (AISTATS 2025) established a generally loose upper bound on the number of linear regions of continuous piecewise-linear (CPWL) Kolmogorov--Arnold networks (KANs), without providing a lower bound. We consider a $T$-layer network with width vector $\mathbf n=(n_0,\ldots,n_T)$, input dimension $n_0=d$, output width $n_T=1$, and at most $k\geq2$ affine segments per edge. Our matching upper and lower bounds determine the tight growth order $Θ_{\mathbf n}(k^{\sum_{\ell=1}^{T}ρ_{\ell-1}})$ in $k$, where $ρ_{\ell-1}:=\min_{0\leq s\leq\ell-1}n_s$. When every hidden width is at least $d$, the tight order simplifies to $Θ_{\mathbf n}(k^{dT})$. The upper bound of \citep{schoots2025relating} is $k^{\sum_{\ell=1}^{T}n_{\ell-1}n_\ell}$. Our upper bound improves it by a factor of $Θ_{\mathbf n}\!\left(k^{\sum_{\ell=1}^{T}n_{\ell-1}n_\ell-\sum_{\ell=1}^{T}ρ_{\ell-1}} \right)$, with strict asymptotic improvement whenever a hidden layer has width greater than one. We further compare CPWL KANs with rectified linear unit (ReLU) networks using dense parameterizations with at most $Λ$ parameters per model and the same fixed nonlinear depth $T$. Counting trainable knot locations and optimizing the widths and $k$, we prove that the maximum region count of CPWL KANs divided by that of ReLU networks is $Θ_{d,T}(Λ^{d\lfloor T/2\rfloor})$ as $Λ\to\infty$, yielding a polynomial advantage for KANs when $T\geq2$. These tight bounds characterize the attainable geometric complexity of CPWL KANs, providing a foundation for understanding their expressive power and guiding architectural design.

Thu 5 NovMachine Learning
The gist
The paper studies how complex certain specialized neural networks called Kolmogorov-Arnold networks (KANs) can be when they use piecewise linear functions. It provides exact mathematical limits on how many distinct linear pieces these networks can create, improving earlier estimates. The authors also compare KANs to common ReLU-based networks and show that KANs can achieve more complexity with the same number of parameters, especially when the network has multiple layers. This characterization helps understand the power of these networks and guides how to build them.
Open → 2611.07973v1

Positional encoding limits length generalization in single-head attention models

A Positional Realizability Theory of Length Generalization in Single-Head Attention

Abstract: Why does length generalization (LG) depend so sharply on the positional encoding (PE)? Relative schemes extrapolate fixed offsets, absolute schemes fix positions, and neither handles growing offsets. We give a realizability theory for a deliberately controlled abstraction, a single positional-only attention head, that derives this well-documented empirical pattern and says precisely where it holds and breaks. The device is a simulation transfer operator $T_L$ that re-evaluates a selected length-$L_0$-realizing positional kernel on the longer grid, with a scalar $α_L \in [0,1]$ read off before any training from $(Φ, t_n)$ alone. Here $α_L$ serves as the computable witness of that taxonomy's derivation, not as a stronger empirical predictor. Our results are a relative-head deep-row characterization for kernels with a strict unique global maximizer (Theorem 2), an absolute/relative duality that derives the pattern (Proposition 4), a learned-absolute underdetermination result, and a static expressivity bound; a sufficiency theorem (Theorem 1) bridges $α_L$ to LG, and the general deterministic case is a precise open conjecture (correcting a tempting but invalid "outside-colspan" argument). Experiments confirm the structure: in the controlled single-head regime $α_L$ exactly separates the taxonomy's generalizing and non-generalizing (PE, task) cells (2625 trained-model measurements; stable at vocab 64/d=128). Together, these results turn the received PE taxonomy into a positional-kernel transfer theorem, and identify the content and depth mechanisms that escape it. The theory concerns the positional channel alone: tasks whose targets depend on token content, including the standard length-generalization suite such as addition and sorting, fall outside Theorem 2.

Fri 30 OctMachine Learning
The gist
Length generalization means how well a model can handle longer inputs than it saw during training. The authors study why the way a model represents positions (positional encoding) affects this ability, focusing on a simple case with one attention head. They find a precise mathematical way to predict when a model’s positional method will work well or fail as input length grows. Their theory explains known behaviors and highlights cases where current intuition breaks down, helping understand limits of absolute and relative position methods. This applies only to certain position-focused tasks, not those depending on token content.
Open → 2610.41218v1

Stability certificates for chains of delayed neural modules hold for any length

Length-Independent Stability and $L_2$-Gain Certificates for Chains of Delayed Neural Modules

Abstract: Networks of repeated neural modules with delayed neighbour coupling need stability guarantees for every number of modules, but delay-dependent criteria certify one network at a time and grow with its size. This paper derives stability and $L_2$-gain certificates of length-independent size for open chains and rings of modules of two alternating types with slope-restricted activations. A local and a transmission delay act independently and are only bounded and continuous. A Lyapunov-Krasovskii functional with fixed integration limits and module-local variables that repeat with the module type reduces the certificate of every chain to one cell of two modules. Local conditions and a single linear matrix inequality of order 96 on this cell, or 104 for the gain, certify global asymptotic stability and a common $L_2$-gain bound for all open chains and all rings with an even number of at least four modules. Up to a margin, the stability inequality is also necessary for the repeated variables to certify every length. All uniform certificates, including certified coupling regions, are verified in exact rational arithmetic. In the numerical study, the certified coupling reaches 92% of an outer bound given by unstable members of the model class for short delays and 79% at the nominal delay bounds. For 16 modules at unit coupling strength, certificates of the same functional computed for that length alone raise the certified delay scale by 2% to 7% in four cases. A single stability problem with 1298 scalar variables thus replaces per-length certification of long chains at little cost.

Sun 18 OctNeural and Evolutionary Computing
The gist
When neural networks are built by linking many identical modules with delays in their connections, it can be hard to prove they will always behave stably no matter how many modules there are. The authors developed a way to test stability using a fixed-size mathematical check that applies to any length of such chains or rings of modules. Their method also provides guarantees about how signals amplify through the network, making it easier to certify large systems without growing complexity. This approach reduces the problem of certifying very long networks to checking a small, repeating unit, verified with exact math.
Open → 2610.25147v1

Minimal neural network widths linked to geometric shapes in math

The minimal width of neural networks and immersions of real projective space

Abstract: We investigate the minimal width of deep neural networks that can approximate all continuous functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ in the topology of uniform convergence on compact subsets. We show that whenever the real projective space $\mathbb{RP}^{n-1}$ admits a smooth immersion into $\mathbb{R}^n$, the minimal width is at least $n+\min\{n,m\}$, assuming that the activation function is continuous and monotone. As a consequence, the minimal width of ReLU networks is $n+m$ whenever $n$ is equal to 2, 3, 4, 7 or 8 and $1\le m \le n$, extending results of Hanin-Sellke and Hwang. This suggests an intriguing connection between the approximation theory of neural networks and differential topology.

Wed 14 OctMachine Learning
The gist
This work looks at how narrow neural networks can be while still being able to approximate any continuous function from multiple inputs to multiple outputs. The authors connect this question to a concept in advanced mathematics called smooth immersions of real projective spaces. They prove that under certain mathematical conditions, the minimal width required depends on the dimensions of the inputs, outputs, and these special geometric shapes. This finding extends previous results and hints at a deeper link between neural network design and mathematical topology.
Open → 2610.20229v1

Gaussian noise models multi-head self-attention spectrum accurately

Gaussian Equivalence for Multi-Head Self-Attention

Abstract: A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.

Wed 7 OctMachine Learning
The gist
Understanding how multi-head self-attention works in neural networks is important for improving AI models. The authors show that by replacing the complex softmax attention step with simpler rescaled scores plus Gaussian noise, the overall behavior of the output remains the same in a mathematical sense. Their work helps separate how different parts of the attention mechanism contribute to the final output and clarifies when different ways of sharing information between attention heads keep the model’s properties stable.
Open → 2610.10033v1

How sparsity and regularization shape learning in overparameterized models

Grokking through the Lens of Minimum-Norm Interpolation

Abstract: Grokking shows that fitting the training data and learning the underlying signal can occur at very different stages. However, existing theories offer limited quantitative insight into how this delayed generalization depends on inductive bias and signal structure. Our work addresses the gap by developing a statistical theory that characterizes how regularization geometry and signal sparsity govern generalization near interpolation. In particular, we focus on the prototypical setting of high-dimensional regression and identify regimes in which sparsity-promoting regularization makes exact interpolation much more accurate than approximate fitting. In strongly overparameterized noiseless problems, we prove a zero--one generalization law and construct a family of convex norms whose interpolators transition from the trivial risk of the all-zero predictor to exact recovery, while keeping the training error equal to $0$. Furthermore, when feature dimension and sample size are proportional, we provide a precise characterization of training and generalization errors along $\ell_r$-regularization paths. This in turn allows us to quantify the generalization gain that remains near interpolation: we show that this gain increases as the norm becomes more sparsity-promoting and as the target becomes sparser, with a sharp drop in generalization reached for noiseless data and $\ell_1$ regularization. Experiments on diagonal linear networks and transformers trained on modular arithmetic demonstrate the generality of our theoretical predictions. Finally, beyond grokking, our work reveals a statistical instability in minimum-norm interpolation: small perturbations in the regularization strength can lead to drastically different generalization, while preserving small training error.

Tue 29 SeptMachine Learning
The gist
Some machine learning models can perfectly fit their training data but only learn the true underlying patterns later, a phenomenon known as grokking. The authors study why this delayed understanding happens by looking at how certain mathematical tools called regularization norms encourage simpler or sparser solutions. They show that when these tools favor sparsity, models can better generalize from data even when perfectly fitting it. Their theory matches experiments with simple and more complex neural networks, highlighting that small changes in regularization can greatly affect how well the model learns beyond just memorizing the data.
Open → 2609.38453v1

Parameter symmetries link to conservation laws in neural network training

On Parameter Symmetries and Conservation Laws in Gradient Flow

Abstract: Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

Mon 28 SeptMachine Learning
The gist
Understanding how neural networks learn involves studying patterns in their parameters during training. The paper clarifies when certain symmetrical changes to network parameters lead to conserved values, drawing from ideas similar to those used in physics. The authors create a geometric framework to explain this connection, especially for complex multilayer networks. They also show how this applies to specific network designs like attention models and polynomial neural networks.
Open → 2609.34549v1

Discovering hidden symmetries in neural network parameters

Discovering Symmetries in Neural Network Parameter Spaces

Abstract: Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

Sun 27 SeptMachine Learning
The gist
Neural networks have many internal settings that affect how well they learn and perform. Finding patterns or symmetries in these settings can help understand how the networks work and how to improve them. The authors created a new mathematical method to automatically find these symmetries, even ones not seen before. They tested their method on different types of networks, including popular transformer models used in AI.
Open → 2609.33527v1

Capacity of single neurons and threshold functions precisely quantified

Boolean threshold functions, neuron capacity, and memory retrieval

Abstract: How much information can a single neuron remember? How many memories can neural networks retrieve without creating false memories? These questions are related to a basic question: how many Boolean threshold functions $f(x)=\operatorname{sgn}(a_0+\langle a,x\rangle)$, $x\in\{-1,1\}^n$, are there? In this paper, we show that the number $T_n$ of distinct Boolean threshold functions is \[ T_n=2\binom{2^n-1}{n}\bigl(1+O(n^{-99})\bigr). \] Equivalently, the capacity of a single threshold neuron is $n^2-\log_2(n!)+1+O(n^{-99})$ bits, improving the $O(n)$ error term in the result of Kahn--Komlós--Szemerédi to $O(n^{-99})$. To prove this, we show that, for $1\le r\le n-1$, and $v_1,\ldots,v_r$ are chosen at random from $\{-1,1\}^n$, \[ \mathbb P\!\left\{ \langle v_1,\ldots,v_r\rangle\cap\{-1,1\}^n =\{\pm v_1,\ldots,\pm v_r\} \right\} =1-O(n^{-99}). \] In the context of the Kanter--Sompolinsky Hamiltonian for memory retrieval, this identifies $r=n-1$ as a sharp threshold, at which, for almost every collection of $r$ memories, the only ground states are these memories and their negatives, confirming a weaker form of the Kalai--Linial--Odlyzko conjecture. It also settles a recent open problem posed by M. Anthony on the specification number of Boolean threshold functions. In addition, we show that, for every $1\le r\le n-1$, \[ \mathbb P\{v_1,\ldots,v_r\text{ are linearly dependent}\} =2\binom r2\,2^{-n}+O\!\left(2^{-n}e^{-cn}\right), \] confirming a conjecture of Kahn--Komlós--Szemerédi.

Thu 24 SeptDiscrete MathematicsNeural and Evolutionary Computing
The gist
This paper addresses how much a single neuron can remember and how many memories neural networks can retrieve without errors. The authors precisely count the number of Boolean threshold functions a neuron can implement, improving previous estimates with much smaller error margins. They also find a sharp boundary where collections of memories can be perfectly recalled without false states, confirming long-standing conjectures. Their work improves understanding of neuron memory capacity and helps clarify mathematical properties of neural networks.
Open → 2609.29756v1

Deep neural networks reach high accuracy by composing simple functions

Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence

Abstract: Deep neural networks approximate functions by composing affine maps with nonlinear activations, but how composition itself creates approximation power is not yet fully understood. We investigate a fundamental mechanism: geometrically weighted sums of iterates of a single scalar generator function. This mechanism underpins the classical tent-map construction of the function \(x - x^2\) and related recursive representations used by Yarotsky, W. E, et al., to analyze the approximation powers of deep neural networks. First, we establish a rigidity theorem: for continuous piecewise linear generators with a finite number of segments, any \(C^3\) function that can be represented in this way is at most quadratic. For non-affine quadratic functions, the geometric factor is at least $1/4$. This result both reveals limitations of the tent-map approach and complements existing methods based on hierarchical bases and recursive polynomial constructions. Second, using an exact remainder identity as guidance, we construct a smooth generator whose iterates yield doubly exponential error decay in total depth for square approximation and, through multiplication modules, for each fixed polynomial. For power series with absolutely summable coefficients on \([-1,1]^d\), distributing depth according to monomial degree yields a uniform approximation error of order \(O(e^{-cL^{1/d}})\) on each interior cube. These findings demonstrate how generator dynamics and remainder estimates govern depth allocation and approximation rates of deep neural networks.

Tue 22 SeptMachine LearningInformation Theory
The gist
This paper looks at how deep neural networks use layers that repeatedly apply simple functions to get better at approximating complex shapes. The authors find that certain simple building blocks can only create limited types of curves, like quadratic ones, showing some limits of these methods. Then they design new building blocks that help neural networks approximate functions much faster, with errors shrinking very quickly as networks go deeper. Their work helps explain how the way neural networks are built controls how well and quickly they learn complex functions.
Open → 2609.25874v1

Neural dynamics must respect language structure for real comprehension

Formal Properties of Language as Constraints on Neural Dynamics

Abstract: What must a neural system be capable of to implement language? Current research annotates stimuli with linguistic variables and tests which electrodes, voxels, or language-model layers predict neural activity. Yet predictive success leaves mechanisms under-constrained. Here, we show that algebraic properties of language specify invariants that mechanisms must preserve: non-associative hierarchical grouping, commutativity, recursive closure, access to substructures, and structured workspace transitions. We term this the Neural Admissibility Program (NAP). Syntactic structure building is analyzed algebraically, with candidate mechanisms offered for each requirement: content-addressable workspace memory, graph-structured transient dynamics scheduling structure-building operations (e.g. stable heteroclinic channels), and a phase-coupled sealing operation recording grouping. Simulations show that a corrected Marcolli-Berwick entropy-optimized binding gate preserves grouping only within a narrow commitment band. As an alternative, we propose a novel neural binding operation we term 'Meld': two constituent populations converge through shared synapses, integrate sublinearly, and saturate. Meld is, to our knowledge, the closest neurally plausible composition law to syntactic Merge. It preserves every NAP invariant, uses known cortical operations, and recovers hierarchical structure at every tested depth and temperature. It predicts that effective population dimensionality separates alternative bracketings and that the composite depends on constituent disagreement. Importantly, the laws decoding bracketing most accurately are a priori inadmissible, showing that decoding accuracy alone cannot adjudicate between mechanisms. By specifying how neural dynamics can remain faithful to linguistic structure, the NAP changes the criterion by which neural implementations of cognition are evaluated.

Sun 13 SeptComputation and Language
The gist
Understanding language requires certain basic rules to be followed by brain processes, such as keeping words grouped properly and recognizing repeated patterns. The authors propose a formal framework called Neural Admissibility Program (NAP) that spells out these essential rules as mathematical properties. They test different neural mechanisms and introduce a new one called Meld, which closely matches how language structures work in the brain. Their work shows that just measuring how well a model predicts brain data is not enough to understand if it truly captures language processing.
Open → 2609.14384v1