Papers for

deep learning engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Blockwise scoring limits accuracy in neural architecture search

When Does Blockwise Additivity Break? Diagnosing and Repairing the Additive Assumption in Distillation-Based Architecture Search

Abstract: Blockwise distillation methods for architecture search, such as DNA, DONNA, LANA and Puzzle, score each candidate replacement block in isolation and assemble a child network by minimizing the sum of per-block scores. We test this additive assumption on DeiT-S and DeiT-B with seven replacement blocks per layer, seven block-latency budgets from 70% to 40%, three fine-tuning seeds and paired bootstrap confidence intervals. (i) At mild compression the additive integer linear program (ILP) is close to a pairwise integer quadratic program (IQP) that models block interactions, but the post-fine-tuning gap widens sharply between the 60% and 55% budgets: at 55%-40% the ILP trails the IQP by 3.3-16.6 top-1 points on ImageNet-100. (ii) We split the KL additivity error into a metric component, caused by the non-linearity of the KL in summed logit changes and computable from cached single-block logits, and a functional component, the score-level effect of non-additive logit changes. The metric component accounts for most of the error on DeiT-S (59-65%), the functional component for most of it on DeiT-B (73-76%). (iii) A hybrid search ranks moves with the logit-sum predictor and measures only the best ones. It needs no dedicated pair measurements, costs 13-27% of the IQP's pair stage, and on ImageNet-100 significantly outperforms the IQP in 7 of 14 settings, with no significant difference in 6 and a significant loss in 1. Further reducing the pre-fine-tuning KL, or scoring after a short fine-tune, gives no consistent gain. On full ImageNet-1k with 1000-class fine-tuning, the sharp widening between 60% and 55% (5.8-17.5 points at 55%-40%), the model-dependent decomposition and the hybrid's advantage (significantly better than the IQP in 13 of 14 settings, no the other) are reproduced.

Thu 5 NovNetworking and Internet ArchitectureNeural and Evolutionary Computing
The gist
When designing smaller, faster neural networks by swapping building blocks, it's common to score each block separately and add those scores to decide the best combination. The authors showed that this 'additive' assumption works well for mild size reductions but breaks down when networks get smaller, causing big accuracy losses. They also separated why this happens and proposed a hybrid approach that selectively checks combinations, improving results without much extra work. Their findings hold true across different models and datasets.
Open → 2611.07487v1

Deep neural networks train better with adjusted edge of chaos settings

Finite-width shift of the edge of chaos in deep neural networks

Abstract: Deep neural networks train best when they are initialised at the edge of chaos, the critical line in the plane of weight and bias variances on which the distance between two inputs neither grows nor shrinks with depth. The line is known for infinitely wide networks; at finite width it moves, and how it moves has been disputed. We treat the logarithm of the distance between two inputs through a network of width $N$ as a random walk and compute its drift and variance to order $1/N$. The edge of chaos moves to a larger weight variance, by $Δσ_w^2=-a_1/(N\,\partial_{σ_w^2}\lnχ_1)$, about $8/N$ for $\tanh$ networks at a bias variance of $0.05$, confirmed by simulations of the exact finite-width recursion. The shift describes a typical network; the average over many networks is dominated by rare networks in which the distance grows, and its critical point moves the other way, which explains why calculations and simulations have disagreed. Backpropagation follows the same random walk, so the shift applies to gradients unchanged. Along the edge of chaos, the information about the similarity of two inputs is constant until the compression over the depth exceeds about one nat, which fixes where to sit on the line. The results give an initialisation rule, Crinit, that places a network on its finite-width edge of chaos without a search. On MNIST and Fashion-MNIST at width $64$ it is more accurate than the standard rules at their usual settings, by a small margin in shallow networks and a large one in deep networks, where most standard rules fail; it trains in fewer steps, and it matches or beats the standard rules with a tuned gain.

Wed 4 NovMachine Learning
The gist
Neural networks learn best when their initial settings are balanced so that small changes in input neither fade away nor explode as they pass through layers. Previous knowledge about this balance was based on infinitely wide networks, but real networks have limited width, which shifts this balance point. The authors found a precise way to calculate this shift, improving how networks are initialized for learning, especially in deeper networks. Their method, called Crinit, helps networks learn faster and more accurately without trial-and-error tuning.
Open → 2611.06582v1

Neural architecture search made faster without training predictors

DIFFNAS: Predictor-Free Neural Architecture Search via Diffusion Model and Zero-Cost Evaluation

Abstract: Neural Architecture Search (NAS) has advanced the automated design of deep networks, but remains computationally expensive due to reliance on model training or performance predictors. We introduce DIFFNAS, a predictor-free NAS framework that leverages a discrete graph diffusion model guided by zero-cost performance proxies. Unlike prior diffusion-based NAS methods that require learned performance predictors, DIFFNAS conditions the generative process on training-free metrics such as MeCo and SWAP, enabling efficient architecture search with minimal computational overhead. Experiments on NAS-Bench-201 show that DIFFNAS finds the optimal architectures for CIFAR-10 and CIFAR-100 (94.37% and 73.51%) without a pretrained predictor, while reducing search cost by up to 17.5x compared to DiffusionNAG. On the MobileNetV3 search space, its accuracy is lower than that of DiffusionNAG. These results highlight the promise of diffusion models combined with zero-cost proxies for scalable and effective architecture search.

Fri 30 OctMachine Learning
The gist
Designing deep neural networks usually takes a lot of time because it needs training many models or using trained predictors to guess performance. The authors present DIFFNAS, a method that skips training predictors by using a diffusion model guided by quick-to-compute zero-cost performance estimates. This makes it much faster to find good network designs, achieving top accuracy on some image tasks while cutting search time substantially. However, in some complex design spaces, its accuracy is not as high as other methods that use predictors.
Open → 2610.41818v1

Training dynamics of deep ResNets simplify to Gaussian mean ODEs

Infinite ResNets are Gaussian Mean ODEs

Abstract: Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.

Fri 30 OctMachine Learning
The gist
Understanding how very deep neural networks learn is difficult because of complex behaviors. The authors show that when ResNet models get infinitely large in width and depth, their training can be described by a simpler mathematical system called a Gaussian Mean ODE. This means the complicated process of training can be seen as a gradient descent on this simpler system, making analysis and perhaps design easier. They also extend this approach to models with multiple experts, showing similar simplifications hold.
Open → 2610.41514v1

Fast layout-aware communication reduces training delays on large gpu clusters

The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch

Abstract: Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.4% of step time in Llama-3.1 70B training at 8K tokens per GPU. Communication kernels impose a second cost: a default NCCL reduce-scatter can slow a concurrent GEMM that would otherwise complete in a single wave by 69%. We present TIDE, a set of zero-copy collectives built on public NCCL 2.31+ APIs. TIDE preserves FSDP2's per-parameter storage and compute kernels while folding layout placement into collective addressing. TIDE-AG delivers shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA, with no SMs used for transport or placement. TIDE-RS consumes per-parameter bf16 gradients directly and reduces them into fp32 shards (a mixed-precision mode that NCCL does not provide) using copy engines and short-lived thread blocks (CTAs) for node-local accumulation and NVSwitch for cross-node reduction. A two-phase pipeline separates gradient reads from network progress; Slot-Reuse optionally stages gradients in vacated all-gather slots. On 64 GPUs, TIDE reduces step time by up to 27.1% relative to Stock FSDP2 and 6.0% relative to FSDP2+SymmMem across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B. It reduces reduce-scatter SM time by factors of 12.5-14.5 and cuts interference with concurrent compute by up to an order of magnitude. Its reduce-scatter buffer remains nearly constant from 16 to 64 GPUs at a fixed per-GPU output size, and its peak memory is up to 4.7 GiB below FSDP2+SymmMem's and, with Slot-Reuse on Wan2.2, below Stock FSDP2's.

Wed 28 OctDistributed, Parallel, and Cluster ComputingMachine LearningPerformance
The gist
Training very large AI models on many GPUs takes time partly because data must be reshaped and communicated inefficiently between parts of the system. The authors show that these reshaping steps and communication slow down the training significantly. They introduce TIDE, a new way to communicate data without extra copying and with better use of system hardware like RDMA and NVSwitch. Their approach speeds up training by up to 27% on large models and reduces delays caused by communication interfering with computation.
Open → 2610.38228v1

Improving accuracy of random neural network models with para-Gaussian methods

Para-Gaussian Approximations of Random Deep Neural Networks

Abstract: It is known that randomly initialized neural networks have Gaussian behaviour in the $n\to \infty$ limit. Unfortunately, this approximation can be poor for finite $n$ and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is \emph{slightly} non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).

Tue 27 OctMachine Learning
The gist
Randomly initialized neural networks are often approximated using Gaussian models, but this can become inaccurate for networks with a limited number of neurons or many layers. The authors propose a new method called the para-Gaussian approximation, which adds a small correction to better capture the network’s behavior. This new approximation uses mathematical tools called Hermite polynomials and allows for recursive calculations that improve predictions about deeper layers. They tested their approach on simpler linear networks and on networks using ReLU activation functions, showing how it can be applied in practice.
Open → 2610.37388v1

Sliding networks keep deep learning stable and diverse features

Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

Abstract: Deep learning models tend to perform best at large widths and depths. The growing interest in scaling laws, especially in the language modeling domain, has motivated procedures for scaling up network widths and depths in order to preserve stability and computational efficiency of training dynamics. Existing depth-scaling approaches force a choice between retaining locally maximal \emph{feature learning} and preserving a form of \emph{feature diversity} in the residual stream arising from random differences between channels. We examine a \textit{joint scaling recipe} that preserves both. With residual width $N_\flat$, hidden width $N_\sharp$, and depth $L$, the key quantity is, by default, the limiting ratio $N_\flat/(N_\sharp L)\toα$, while correlated initialization (Half Copy) can partially relax the resulting shape constraints. Theory and residual-MLP experiments support convergence at fixed positive $α$ to an SDE limit with locally maximal feature learning. Hidden computations can then be redistributed between width and depth while preserving this limit, like turning melody into harmony and back. Language-model experiments nevertheless show performance differences under regrouping at fixed parameter count. These results distinguish approximating a limit (ODE or SDE) from scaling for better performance.

Sat 24 OctMachine Learning
The gist
Deep learning models work better when they are bigger and deeper, but making them bigger often forces a choice: either learn good features or keep a variety of different features. The authors propose a way to scale deep networks that keeps both good feature learning and diverse features at the same time. They test their approach with theory and experiments on smaller networks and language models, showing it balances learning well but can affect performance differently when reshaped. Their work helps understand how network size and shape affect learning quality and diversity.
Open → 2610.33840v1

Backward coupling improves cross-layer learning in deep neural networks

Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

Abstract: During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.

Fri 23 OctMachine Learning
The gist
Deep neural networks learn by passing information forward and adjusting connections backward during training. The paper studies how feedback from later layers can reshape earlier layers’ features, something absent in older layer-wise methods. The authors develop a theory and algorithm that captures this backward influence through spectral (frequency-like) modes and test it on image recognition tasks. Their method improves feature learning and prediction accuracy compared to previous forward-only approaches.
Open → 2610.32849v1

Infinite width and depth limits match during neural network training

Width and Depth Limits Commute During Training

Abstract: Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$. In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same $1/\sqrt{depth}$ residual-branch scaling (Depth-$μ$P) and the $μ$P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in $L^2$ along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding $L^2$ convergence does not hold in general, but we establish the weaker convergence in probability.

Thu 22 OctMachine Learning
The gist
When building very large neural networks, it matters whether you make them wider (more neurons per layer) or deeper (more layers). This paper shows that, for certain types of networks and scaling rules, the final behavior is the same whether you make them infinitely wide first, infinitely deep first, or both at once. The authors prove this holds not only when the networks start training but also during training, as long as some precautions like gradient clipping are used. Without those precautions, the convergence is weaker but still happens in a probabilistic sense.
Open → 2610.30729v1

Deep network training converges better with noise and wider layers

Global Convergence in Deep Networks via the Second Law of Thermodynamics

Abstract: Existing convergence analyses of finite-width overparameterized deep neural networks under standard parameterization often rely on parameters remaining close to their initialization and typically neglect the role of noise, despite empirical evidence that stochasticity can improve optimization and generalization. To bridge this gap, we study $L_2$-regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations. We introduce a change-of-measure inequality, inspired by the second law of thermodynamics, that enables control of key properties of the training dynamics under the time-evolving parameter distribution. Leveraging a sharpened lower bound on the minimum eigenvalue of the Neural Tangent Kernel (NTK) at initialization, we show that injected noise helps preserve NTK stability with high probability without confining the weights to a small neighborhood of their initialization. As a result, we derive a non-asymptotic upper bound on the expected training loss. Our analysis yields explicit conditions on width and noise temperature that guarantee convergence to a finite loss floor. Finally, we experimentally show the network Jacobian can change from initialization, departing the commonly analyzed lazy regime.

Sat 17 OctMachine Learning
The gist
Training very large deep neural networks usually assumes the parameters don’t move far from where they start, but in reality, noise during training helps improve performance. The authors studied a training process that adds controlled noise to the network parameters and found that this noise helps keep key features stable even as the parameters change significantly. This stability allows the network to learn better without being stuck close to its starting state. Their math shows how wide the network needs to be and how much noise to add for training to reliably improve.
Open → 2610.24775v1

LeMA scales Levenberg-Marquardt training for billion parameter models

Scalable Levenberg-Marquardt Training Framework for Deep Learning

Abstract: The Levenberg-Marquardt (LMA) algorithm is a widely used nonlinear least-squares solver known for its rapid convergence. For small-scale neural networks, LMA provides better convergence and architectural robustness than first-order optimizers like Adam. However, LMA imposes prohibitive memory and computational costs. These costs are primarily from the Jacobian matrix, whose dimensions scale with both the number of samples and parameters. This bottleneck prevents its use in large-scale neural networks. Recent efforts scale LMA using preconditioned conjugate gradient (PCG) solvers to avoid materializing the Jacobian. However, PCG only approximates the solution, requiring many network propagations and a task-specific preconditioner. Other studies optimize LMA to reduce costs, but they restrict their methods to specific tasks or network architectures. As a result, LMA remains impractical for training general large-scale neural networks. In this paper, we propose LeMA, a distributed LMA algorithm for large-scale neural network training on multi-GPU platforms. Our contributions are twofold: First, an exact dual formulation of LMA removes the memory dependence on the parameter dimension. Second, a Jacobian sharding strategy computes Jacobian shards on the fly and in parallel to avoid materializing the full Jacobian. For large-scale neural networks, LeMA reduces the memory footprint of LMA from quadratic in the number of parameters to quadratic in the batch size. This makes exact LMA steps feasible for billion-parameter models without approximate PCG solves. Furthermore, for a convolutional neural network with 1,000,022,632 parameters on an 8-GPU node, LeMA achieves a 7.97x speedup over a single GPU, corresponding to a 99.7% strong scaling efficiency.

Fri 16 OctMachine LearningDistributed, Parallel, and Cluster Computing
The gist
Training big neural networks can be slow and require a lot of computer memory, especially with some older powerful math methods. The paper shows how to make one such method, called Levenberg-Marquardt, work efficiently even on very large networks using multiple GPUs. The authors designed a way to split and handle data so computers don’t get overwhelmed by memory needs. This speeds up training significantly while still keeping accuracy, making it possible to use this method on huge neural networks.
Open → 2610.22925v1

Lossless compression method reduces GPU memory for AI training

LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

Abstract: Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT), a GPU execution framework designed for this repeated encode-decode regime. LCT exploits the natural redundancy in floating point data to reduce storage for weights, activations and optimizer states. Highly optimized GPU kernels implement Huffman coding with precomputed codebooks for low compression and decompression overhead, essential for training. Our analysis characterizes LCT's storage costs and adaptation to magnitude shifts. LCT achieves a compression ratio of up to $1.455\times$, and is demonstrated in training and finetuning settings. On a 162M-parameter NanoGPT, LCT reduces memory usage by $24.6\%$ for a $7.2\%$ step-time increase. In Qwen3-4B fine-tuning, LCT saves up to $23.8\%$ of peak memory, enabling a $3.5\times$ larger context length in training on the same GPU.

Thu 15 OctMachine Learning
The gist
Training AI models uses a lot of computer memory, partly because data needs to be stored and repeatedly compressed and decompressed. The authors created a new method called Losslessly Compressed Tensors (LCT) that stores this data in a smaller space without losing any information. They designed it to work fast on GPUs by using efficient coding techniques. Their method cuts memory use by about a quarter but slows training steps slightly. This allows bigger AI models or longer training sequences on the same hardware.
Open → 2610.22179v1

Depth and width shape neural network features in precise formula

A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

Abstract: How depth and width of neural networks shape learned representations is usually studied only after training. We show that, in a tractable feature-learning setting, this dependence can instead be obtained in closed form. Using neural collapse as a probe of representation geometry, we express NC1--NC3 directly in terms of the learned feature kernel, extending an identity known for NC1 to the angular part of NC2 and to NC3 under a ridge readout. For deep linear networks with exchangeable class structure, the resulting kernel equations yield a geometric law, exact within the theory, for the layerwise evolution of within-class variability. At large width, the accumulated compression becomes a function of the depth-to-width ratio $L/N$, with its rate determined self-consistently by the data, prior and architecture. Numerical experiments support the derived dependence on depth, width and signal-to-noise ratio. The result is a representation-level scaling law derived from feature-learning theory rather than fitted after training.

Thu 15 OctMachine Learning
The gist
How the depth (number of layers) and width (number of neurons per layer) of neural networks affect their learned features is usually understood after training. The authors show in a simplified, mathematically tractable model how this relationship can be expressed exactly before training. They link the geometry of learned features, through a phenomenon called neural collapse, directly to network architecture and data properties. Their results reveal that how features compress inside the network depends on the ratio of depth to width and matches experimental observations.
Open → 2610.21820v1

Quantizing language model memories cuts storage by three quarters

N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models

Abstract: Conditional memory expands language-model capacity through selective lookup, but the full embedding table must remain available for every possible input. We introduce Conditional Program (CP), a compressed representation that reconstructs only the rows an input requests. CP represents each row by a short code and a local scale, with a reconstruction program shared across the table. We instantiate this representation with a Gaussian trellis, norm restoration and independently decodable row records. CP's two-bit code lowers output KL by 41--63\% relative to 2-bit round-to-nearest. Analysis across three trained n-gram memory families shows that Gaussian-like coordinate statistics coexist with learned relationships among rows; at matched storage budgets, the shared trellis outperforms the fitted codebooks and low-rank residual variants we test. For DeepSeek, it reduces table storage from 203GB to 51GB, leaving room for larger workloads when the table is kept on GPU.

Thu 15 OctMachine LearningComputation and Language
The gist
Language models use large tables to remember context, but these tables can be huge and hard to store efficiently. The paper introduces a new way to compress these tables, storing only what’s needed for each input with fewer bits per entry. Their method uses a shared program to reconstruct specific parts, reducing storage size significantly without losing much accuracy. This lets bigger models run on devices like GPUs without needing as much memory.
Open → 2610.21810v1

Layer normalization alters neural network width and approximation needs

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Abstract: Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-$N$ one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width $N+1$, while every continuous nonconstant activation admits width $N+r$ with error $O_f(r^{-1})$. For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width $N+1$; general continuous activations admit width $N+r$ with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths $N+2$ and $N+2r$, respectively. We prove that $N+1$ for RMSNorm and $N+2$ for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.

Thu 15 OctMachine Learning
The gist
Layer normalization and a related method called RMSNorm change how neural networks behave by affecting their internal calculations during use. The authors studied how these normalizations influence a network's ability to mimic other simpler networks of fixed size. They found specific minimal sizes (widths) needed for networks using these normalization methods to approximate others, depending on details like activation functions. Their results provide formulas and proofs about these requirements and confirmed the findings through experiments.
Open → 2610.21527v1

New way to measure neural network accuracy without size effects

Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes

Abstract: Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space $Θ\subseteq \mathbb{R}^D$, causing relative entropy to scale with network dimension $D$. For modern overparameterized neural networks where $D \gg n$, this parameter-counting dependency yields vacuous generalization bounds ($> 1{,}000\%$). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions $\mathcal{M}_1(\mathcal{Z})$ via a \textbf{Dirichlet Process (DP)}, $\mathcal{DP}(α, F_0)$, inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count $D$, width, and depth. On intrinsic $d^*$-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate $\mathcal{O}(n^{-1/(d^*+2)} \sqrt{\ln n})$. Across vision architectures scaling up to 4.2 million parameters ($D = 7{,}850$ to $D = 4{,}203{,}522$) on the full MNIST benchmark, while classical weight-space bounds scale past $940\%$ (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at \textbf{$8.8\%$--$9.4\%$}.

Thu 15 OctMachine Learning
The gist
Traditional ways to predict how well deep neural networks will perform get worse when the networks become very large because they depend heavily on counting the number of parameters. The authors propose measuring generalization by focusing on the data distribution instead of the network parameters, using a mathematical tool called the Dirichlet Process. This approach removes dependence on network size and provides stable, meaningful guarantees even for networks with millions of parameters. Their method was tested on popular image recognition tasks and showed much tighter and practical bounds compared to older methods.
Open → 2610.21340v1

Polynomial activation guides deep network design and training success

Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

Abstract: What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + λz 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.

Wed 14 OctMachine LearningArtificial Intelligence
The gist
Training very deep neural networks can be difficult because of how their error landscapes look. This paper shows that using a special kind of activation function—a polynomial that includes a cubic term—helps even very deep networks train well without extra tricks. The authors studied a small version of this network where they could solve the training problem exactly, and used that understanding to explain why the deep versions work so well. Their work offers a practical way to design, initialize, and train deep networks more effectively.
Open → 2610.20069v1

Continuous Thought Machine traces image classifications along a fixed pattern

Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine

Abstract: The Continuous Thought Machine (CTM) is a recurrent network that attends to inputs and makes predictions based on the synchronization between its neurons. On tasks spanning many domains, it displays emergent thought processes that have not been explained. We report a number of findings about how the CTM classifies images, focusing on ImageNet. Each thinking step is an attracting but strongly non-normal map exhibiting balanced amplification (Murphy and Miller, 2009); while attraction is already present at CTM initialization, training highly promotes non-normality. By Banach's theorem, the neural state has a locally unique attracting fixed point, which moves at every step, and the thought trajectory is the curve these fixed points trace, followed one tick behind. Most of that curve's motion is shared across images, and visible as a previously reported traveling wave of activity across neurons. The attention schedule along the thought is largely generic: shuffling queries across images at every tick costs only 7 percent of classification accuracy, while image-independent increments predict the next query with R^2 = 0.98 (held out). Our results suggest that the CTM has found a single strategy to process all ImageNet images.

Tue 13 OctMachine LearningComputer Vision and Pattern RecognitionNeural and Evolutionary Computing
The gist
Classifying images is a complex task, and this paper looks at how a special type of neural network called the Continuous Thought Machine (CTM) does it. The network processes images step-by-step by moving through a series of stable states, like following a path of fixed points that change over time. Most of this processing path is shared for all images, which means the CTM uses a common strategy to understand different pictures. The authors also found that swapping parts of the network’s attention does not harm accuracy much, showing this process is very stable and generic.
Open → 2610.19220v1

Extreme sparsity breaks neural network connectivity and guided fixes improve training

When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Abstract: Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies. At extreme sparsity ($\ge 99\%$), this dynamic tends to destroy input-output information pathways, and performance with standard magnitude/random updates degrades sharply. We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory. Guided by this reachability-based structure, we bias pruning toward non-functional edges and regrowth toward functionally consistent candidates. We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST. On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to $+14.4$ points at $99.9\%$ sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at $99.99\%$ sparsity while unguided random regrowth degrades sharply. These results cast extreme-sparse training as a connectivity-preserving dynamical process. Our implementation is publicly available at https://github.com/mzjaworski/ea_dst.

Mon 12 OctMachine Learning
The gist
Training very sparse neural networks, where most connections are removed, tends to break the pathways that carry information from inputs to outputs, causing performance to drop sharply. The authors studied how these connection paths form and disappear in sparse networks and developed a way to track and protect them during training. By guiding which connections to prune and regrow based on their role in the network, they restored accuracy on image classification tasks even at extremely high sparsity levels. This approach treats sparse training as a process of preserving key connectivity in the network.
Open → 2610.17425v1

DeepGroundRepair improves reliability of deep learning code generation

DeepGroundRepair: From API-Grounded Generation to Shape-Aware Repair for Deep Learning Programs

Abstract: Deep learning programs are fundamental to modern AI applications, and automatically generating them from natural language requirements can substantially reduce DL development effort. Large language models have demonstrated promising capabilities for this direction. However, generated DL programs often remain unreliable because they must satisfy domain specific API usage constraints and tensor shape dependencies, violations of which can cause API level syntactic issues and runtime failures. Existing retrieval augmented generation and feedback guided code repair methods do not adequately address these DL specific constraints. In this paper, we propose DeepGroundRepair, a three stage framework that establishes a generation localization repair workflow to improve the reliability of LLM-generated DL programs. Its key idea is to progressively narrow the generation and repair spaces using DL specific knowledge and runtime tensor information. Specifically, it grounds generation in task-relevant API knowledge, localizes the upstream causes of tensor shape mismatch failures, and uses runtime tensor shapes to guide parameter repair. We evaluate DeepGroundRepair on the 100 model-level DL programming tasks in DeepEval using six open-weight LLMs across TensorFlow, PyTorch, and PaddlePaddle. We compare it with four baselines: two generation-only methods and their counterparts incorporating traceback guided self debugging. The results show that, relative to these baselines, DeepGroundRepair reduces the combined number of API-level syntactic issues by an average of 5.4 per 10 program files and achieves an average relative improvement of 85.3% in code executability. Compared with the two self debugging baselines, it improves semantic consistency by an average of 18.1% and reduces token cost by an average of 28.4%.

Mon 12 OctSoftware Engineering
The gist
Writing computer programs for deep learning is tricky because they have to use special rules and work with data shapes correctly. The authors created DeepGroundRepair, which makes AI models that write code better by teaching them these rules and checking for mistakes step by step. This process helps fix errors caused by wrong use of commands and data sizes, making the generated programs work more often and cost less to fix.
Open → 2610.16897v1

Reducing activation bias slows learning but improves final AI performance

Rethinking Activation Functions: Reducing Bias Can Improve Learning

Abstract: Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.

Sat 10 OctMachine LearningArtificial Intelligence
The gist
Many popular activation functions in AI models tend to produce mostly positive outputs, causing a bias that affects how the model learns. The authors found that this bias helps models learn quickly at the start but leads to unstable learning and worse final results. When the bias is reduced, learning is slower initially but becomes smoother and ends with better overall performance. Tests on large vision and language AI models confirmed this pattern.
Open → 2610.14500v1

Memory budget increases can unexpectedly break neural network training

When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization

Abstract: Dynamic Tensor Rematerialization (DTR) trains neural networks under a memory budget by evicting tensors and recomputing them on demand. A natural expectation is that a budget large enough to complete training stays sufficient when it is increased. Using DTR's reference simulator, simrd, to replay the public DTR training traces, we show that this expectation fails. We call the exceptions feasibility holes: memory budgets at which an eviction policy runs out of memory although both a smaller and a larger budget complete, a form of non-monotone feasibility. Under a written sweep protocol with dated amendments, a 0.01-step budget grid shows no such hole for DTR on any of six confirmatory traces, although unresolved timeouts prevent it from establishing monotonicity. Finer sampling finds holes on InceptionV4 (three regions) and U-Net (two regions), in addition to one on the ResNet-32 development trace. Every hole is reproduced by three repeats and by unmodified simrd. For one InceptionV4 pair (0.2343 versus 0.235 of peak memory) we trace the failure to individual allocations. The larger budget avoids one eviction at an operator where the smaller budget was 173,479 bytes short. The extra data it carries forward forces a later eviction of a tensor needed three operators afterwards, and rebuilding that tensor triggers a recomputation cascade that cannot fit. Keeping that one tensor resident under the same budget prevents the failure. The two simple score changes we tested do not remove holes across traces; both add failing budgets on some traces. All results are from simulation, with no allocator model; behaviour on GPU hardware is untested.

Fri 9 OctMachine LearningDistributed, Parallel, and Cluster Computing
The gist
Training big AI models uses a lot of computer memory, so some systems save space by deleting and recalculating data when needed. The authors found that sometimes having slightly more memory can actually cause training to fail, even when a little less memory worked fine before. They studied this unexpected problem, called feasibility holes, by simulating how memory is used during training of popular neural networks. Their findings reveal a tricky behavior in memory management that was not previously known and show it can cause training failures in specific cases.
Open → 2610.14369v1

Stable learning rate scaling enables deep neural network training

On the Depth Transfer of Neural Optimizers via L-Smoothness

Abstract: Scaling deep learning models efficiently requires predictable performance transfer across compute regimes. Establishing reliable hyperparameter transfer allows optimal training dynamics and structural choices to be discovered on low-cost proxy models, avoiding catastrophic failures during massive-scale training. Existing depth-transfer analyses are architecture-specific and heuristic: hyperparameter transfer is only justified through toy blocks, and both rely on infinite-depth dynamics, a specific initialization, and feature-learning criteria. Building on $L$-smoothness framework of Xu et al. (2026), we understand hyperparameter transfer as how do the Lipschitz and $L$-smoothness constants of the loss scale with the depth $K$? We show that inverse-depth residual branch scaling $α=Θ(1/K)$ is sufficient to make both constants depth-independent at fixed width, uniformly over the bounded parameter domain and for arbitrary $p\to q$ geometries. Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks (including branches of block depth $\geq 2$ such as transformer and ResNet blocks) without assumptions on initialization or correlations between layer weights. Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm. This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA. Learning-rate sweeps on GPT-2 (depths 2--128) and CIFAR-100 ResNets (8--32 residual blocks) confirm that Muon and MOGA exhibit stable learning-rate transfer under these prescriptions, with a common optimal rate across a 64-fold increase in depth.

Fri 9 OctMachine Learning
The gist
Training very deep neural networks can be tricky because settings that work for small models often fail when the model gets bigger. The authors studied how certain mathematical properties of the model's loss change as the network gets deeper. They found that scaling parts of the network in a certain way keeps these properties stable, which means learning rates and other training settings can be transferred safely from smaller to much larger networks. Their experiments with popular models confirmed that this approach helps keep training efficient and stable over a wide range of depths.
Open → 2610.14344v1

SlackSwap improves training speed with smarter GPU memory use

SlackSwap: Bandwidth-Aware Activation Swapping and Recomputation for Memory-Efficient DNN Training

Abstract: Swapping saved activations to host memory once enabled GPUs to train models that exceeded their device-memory capacity when compute throughput and host-link bandwidth were roughly balanced. For example, on a 2015 Titan X, PCIe bandwidth was sufficient to offload all VGG-16 activations during the forward pass. This balance no longer holds. From the Titan X to the H100, peak training throughput increased by 141x, whereas host-link bandwidth improved by only 3.9x. On an A100, the host link can hide transfers for only 11-38% of the activations in the transformer models we study. Existing swapping approaches, such as vDNN, SuperNeurons, and SwapAdvisor, offload tensors according to memory requirements without explicitly respecting the available link-time budget, resulting in transfer-induced stalls; block checkpointing, in contrast, leaves the host link underutilized. We present SlackSwap, which treats host-link time as an explicit resource budget. Using a single profiled iteration, SlackSwap allocates the forward-pass link budget to tensors that are most expensive to recompute, resolves the remaining memory deficit by recomputing the least expensive tensors, and schedules FIFO offloads and earliest-deadline-first prefetches without per-layer synchronization barriers. In trace-driven simulations of BERT-Large, GPT-2 XL, and a Llama-7B-style model on an A100, SlackSwap retains 96-100% of unlimited-memory GPU throughput while using half the activation memory, achieving 1.10-1.11x the throughput of the best baseline. Its plans are within 0.5% of a genetic search over 6,600-14,000 candidate plans, while requiring less than 0.4 seconds to generate.

Fri 9 OctDistributed, Parallel, and Cluster Computing
The gist
Training large AI models on GPUs can run into memory limits, forcing slow data swaps between GPU and host memory. The authors show that past swapping methods don’t perfectly balance data transfers and recomputations with the link speeds available today. They introduce SlackSwap, a smarter way to manage these swaps by prioritizing which data to transfer or recompute based on bandwidth constraints. This approach keeps training fast while using less GPU memory, nearly matching the speed of setups with unlimited memory.
Open → 2610.13003v1

Targeted muon method improves training for language and vision AI

Spectrally Targeted Muon

Abstract: The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models. It remains unclear, however, whether this success comes from amplifying small singular directions that gradient descent neglects or from suppressing large, degenerate directions that disrupt training. We introduce Spectrally Targeted Muon, which orthogonalizes only the singular values above or below a threshold $τ$, so that varying $τ$ interpolates between normalized SGD and Muon. It isolates the relevant singular subspaces with projections computed by Newton-Schulz iteration on a shifted Gram matrix, so no SVD is needed. We evaluate these variants on the CIFAR-10 and NanoGPT speedruns, tracking the effective rank of gradient, update, and weight matrices and a new metric, the alignment of updates with the tangent space of the weight matrix's isospectral manifold. We find three things. First, the small singular values of the momentum are not noise. Orthogonalizing everything except the few largest singular values of each matrix nearly matches Muon while touching only a small fraction of the momentum, whereas orthogonalizing only the top falls well short even though it holds almost all of it. On language models every momentum singular value is far below one, so targeted orthogonalization can only amplify, and Muon wins by making directions that are too small to train on at their raw scale trainable. Second, shrinking the largest singular values is what keeps the parameter spectrum flat; this is cheap, and it is not what drives the loss. Third, AdamW differs from Muon mainly in how slowly it builds structure, which explains its slower start and why warmup helps AdamW but only hurts Muon.

Wed 7 OctMachine Learning
The gist
Training AI models can be tricky because some directions that help learning are very small and often ignored. The paper introduces a way to focus on certain parts of these learning steps by only adjusting the big or small parts of updates, instead of all parts. This new method helps the model learn better and faster, especially for language models where many update parts are very small at first. The authors also compare this method to others like AdamW and show it builds useful learning directions faster.
Open → 2610.10965v1

Gpu-based method speeds up topological analysis of images dramatically

GPU-Accelerated Computation of Persistent Homology for Topological Analysis of Image Data

Abstract: In recent years, persistent homology has seen rapid adoption in deep learning, yet its computation remains a major bottleneck in network training. This paper introduces TopoGPU, a GPU streaming pipeline that computes persistence diagrams of cubical complexes induced by 2D and 3D images. TopoGPU streams the input image chunk by chunk, processing each chunk with massively parallel GPU kernels on a grid of GPU blocks; the resulting boundary relations are accumulated in host memory, where the CPU performs the boundary matrix reduction. TopoGPU introduces a stratification-aware discrete Morse matching that provably preserves persistent homology under streaming, together with a parallel topological sorting algorithm and a parallel V-path parity algorithm for deriving Morse boundaries on the GPU. TopoGPU outperforms Cubical Ripser, a state-of-the-art method for persistent homology computation, on every benchmark evaluated, achieving an average end-to-end speedup of 53.24x and a maximum of 198.01x. We further integrate TopoGPU into a topology-preserving deep network, demonstrating that it substantially reduces the cost of persistent homology computation during network training. TopoGPU is open source, with pre-built binaries, Google Colab notebooks, and Docker images available at the project's GitHub page: https://github.com/seravee08/GPU-Computation-of-Persistent-Homology-for-Image-Data.

Wed 7 OctComputer Vision and Pattern Recognition
The gist
Topological analysis helps computers understand the shape and structure of data, but it can be very slow, especially for image data used in deep learning. The authors present TopoGPU, a system that uses graphics processors (GPUs) to speed up these calculations by breaking images into chunks and processing them in parallel. Their approach carefully preserves the important topological features during streaming, resulting in much faster computation than previous tools. This makes it easier and faster to include topological analysis in training deep neural networks.
Open → 2610.10959v1

Velocity scaling improves image generation quality by correcting time lag

Velocity Scaling in Flow Matching

Abstract: Scaling a learned flow-matching velocity field $v_θ$ by a gain $γ(t)$ was recently shown to greatly improve generation quality. Prior work argued that velocity fields trained with mean-squared error (MSE) systematically underestimate velocity magnitude and that scaling corrects this error. We show that MSE training does not create a velocity-magnitude deficit. We find instead that velocity scaling reduces population time lag: sampled states at model time $t$ resemble training states from an earlier time. Velocity scaling and moving model time back are two ways to address this population time lag. Across architectures and model sizes, measuring population time lag and using it to select a gain greatly improves generation quality, reducing FID from 28.0 to 12.2 (estimated by linear interpolation between FID measurements at neighboring gains) on ImageNet-256 at NFE 25 without guidance.

Wed 7 OctComputer Vision and Pattern Recognition
The gist
Creating images using a model that simulates changing states over time can be tricky because the model's imagined progress can lag behind real training progress. The authors show that adjusting how fast the changes happen helps fix this lag, making generated images much better. They found that the common idea that the model underestimates change size is incorrect. Instead, changing the speed helps the model match the right stage of image creation, leading to clearer pictures.
Open → 2610.10823v1

Hessian eigenvalue patterns explained by hidden symmetry in deep learning

Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

Abstract: Hessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated outliers. This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference. Modifications, including changes to the architecture, data distribution, or parameter metric, expose a nearby reference configuration whose Hessian exhibits rich invariances-ones not accounted for by weight symmetries. There, symmetry enables a precise description of the spectra, forcing high-dimensional kernels and eigenvalues of large multiplicity. Returning to the original configuration breaks the Hessian symmetry and thereby produces the observed hierarchy of clusters and outliers. The framework is developed in some generality, with a detailed analysis of three-layer ReLU networks and applications to convolutional, graph, and transformer models, as well as to the NTK. The same mechanism is further shown to yield analogous spectral structures in layerwise Hessians and the Gauss-Newton matrix.

Wed 7 OctMachine Learning
The gist
Deep learning models have complex structures, and a key to understanding their behavior lies in something called the Hessian matrix. The authors found that the pattern of Hessian eigenvalues can be explained by revealing a hidden, highly symmetric version of the model. This symmetry creates clear groups of eigenvalues, and breaking the symmetry leads to the clusters and outliers seen in real trained models. Their framework applies broadly to various neural network types, including convolutional and transformer models.
Open → 2610.09919v1

Training vision models improves with better choice of Inception crop scales

You May Be Running the Wrong Inception Crop

Abstract: A decade after its inception, Inception crop has become the standard crop-based data augmentation method for training deep vision models. Not only is its practice of uniformly sampling crop scale and aspect ratio widely adopted, but also its lower and upper bounds, with the scale lower bound being the sole exception that is sometimes tuned. It is therefore surprising that the standard implementation in the TensorFlow / JAX ecosystem samples crop scale with probability density function $f(A) \propto \frac{1}{\sqrt{A}}$ unlike the PyTorch counterpart, which follows the original description. Motivated by this discovery, we train 522 ViT-S/16 models on the ImageNet-1k dataset with various training budgets and crop scale distributions. We reach $78.78\pm0.09$ top-1 val. accuracy with 90 epochs of training budget and find that 1. Higher training budget requires stronger augmentation; 2. Lower tail of the distribution of the crop scale determines the augmentation strength of Inception crop; 3. Models trained with higher training budget exhibit sparser saliency, regardless of the crop scale distribution or weight decay. Based on 2. we revisit the performance of Beta crop, whose softer cutoff allows it to optimize model performance across training budgets with less compromise. We replicate 1. and 3. with Scion optimizer in addition to AdamW, suggesting that the results may be general.

Fri 2 OctMachine Learning
The gist
The paper reveals that a commonly used method for preparing images during training is often implemented differently across popular software, which can affect model performance. The authors trained many vision transformer models on a large image dataset and found that the exact way image crops are sampled matters, especially for longer training. They show that stronger and more flexible cropping strategies help models learn better, leading to improved accuracy and clearer model focus. Their findings hold true across different optimization methods, suggesting broad relevance.
Open → 2610.04147v1

Residual networks use dynamic patterns to route information internally

Do ResNets Route? Sparse Interaction Experts in Residual Networks

Abstract: Residual networks execute every block for every input, yet their functional contributions need not be input independent. We formulate a trained ResNet as a set function over binary residual-branch masks and apply Möbius inversion to decompose its output exactly into individual residual corrections and higher-order interactions. For smooth residual stacks, we show that each fixed $k$-way interaction scales as $\mathcal{O}(λ^k)$ under residual scaling. Exhaustive analysis of ImageNet-pretrained ResNet-18 and ResNet-34 reveals that interaction mass peaks at orders five and ten, respectively, rather than at low orders. Reducing the residual scale shifts both spectra toward lower orders, but also changes model predictions. The interaction coefficients are concentrated in magnitude but not hard sparse, and prediction-preserving sparsity weakens with depth. Crucially, the dominant interactions vary across inputs and predicted classes around a shared global core, while their overall complexity changes little with sample difficulty. These results show that dense ResNets implement an implicit form of soft routing: every block is executed, but different inputs rely on different residual interaction experts. Routing can therefore emerge at the level of functional contribution without an explicit router or sparse execution.

Fri 2 OctMachine Learning
The gist
Residual networks are a type of deep learning model that always apply every step of their process, but this paper finds that different steps contribute differently depending on the input. The authors analyze how these networks combine various parts and show that the importance of these parts varies and interacts in complex ways. Unlike traditional routing which skips parts, these networks softly route by weighting contributions from different sections without turning any off completely. This reveals a new understanding of how these models flexibly handle diverse inputs internally.
Open → 2610.02907v1

Dynamic correction improves efficient matrix multiplication in deep learning

DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNs

Abstract: Large-scale foundation models achieve strong performance across diverse tasks, but their size makes inference costly, largely due to dense matrix multiplications. Prior work reduces this cost by replacing dense weight matrices with efficient structured forms such as low-rank factorizations. However, these methods approximate weights rather than the output activations that determine inference accuracy. Consequently, small weight-space errors can be amplified by input activations, producing large output errors. In this work, we propose DyRA, an input-adaptive method that improves structured matrix multiplication approximation by correcting residual output errors during inference. We show that matrix multiplication can be approximated more effectively by directly optimizing low-rank factors of the output. DyRA builds on this insight by dynamically approximating and correcting the output error introduced by structured weight approximations. This combines efficient structured computation with input-dependent correction, yielding a more faithful approximation of full matrix multiplication under the same computational budget. Across vision, speech, and language models, DyRA consistently improves the accuracy-efficiency trade-off over structured weight approximations alone. Notably, DyRA achieves a 1.5$\times$ end-to-end GPU speedup for DINOv3 while reducing accuracy degradation by more than 3$\times$ relative to weight-only baselines.

Fri 2 OctMachine LearningArtificial Intelligence
The gist
Running large AI models can be very slow and costly because they need to do lots of math with big tables of numbers. The authors noticed that earlier methods made these tables simpler but sometimes caused big mistakes. They created DyRA, a smarter way that not only simplifies the math but also fixes errors on the fly using the actual input data. This approach makes the operations faster and more accurate across different AI tasks like vision and language.
Open → 2610.02882v1

SoftServe improves deep learning training with scalable quasi-Newton methods

SoftServe: A Scalable Quasi-Newton Method for Deep Learning

Abstract: Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.

Thu 1 OctMachine LearningArtificial Intelligence
The gist
Optimizing deep neural networks can be very hard because they have many parameters and their math is complicated. The authors created SoftServe, a new way to improve training by estimating curvature to guide updates better, even when the usual assumptions don’t hold. It uses efficient math tricks that work well with GPUs and can handle very large networks. This approach often achieves better performance than popular existing methods on difficult learning tasks.
Open → 2610.02182v1

Less cuts neural architecture search time to minutes with strong accuracy

LESS: Lightweight Evolutionary Supernet Search in Minutes

Abstract: Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of candidate comparisons. Training-free methods reduce evaluation cost by replacing learned task feedback with proxy signals measured at initialization. We introduce LESS (Lightweight Evolutionary Supernet Search), a data-driven method that combines a brief fair hard-path warm-up with discrete search under a single CMA-ES distribution. Each proposal is evaluated as its decoded hard genotype after six candidate-conditioned supernet updates. On NAS-Bench-201, LESS achieves \(93.189\pm0.467\%\) CIFAR-10 test accuracy in 409.1 seconds, coming within 0.04 percentage points of FairNAS using approximately \(1/24\) of its source-reported search time. Matched controls show that calibration improves selected validation accuracy by \(0.577\) percentage points while changing best-visited accuracy by only \(0.054\) points, indicating that its primary effect is to reduce selection regret. The frozen configuration transfers without tuning to CIFAR-100 and ImageNet16-120 with \(69.615\pm1.139\%\) and \(43.720\pm1.697\%\) accuracy. Applied without tuning to the larger DARTS space, LESS achieves \(96.95\pm0.14\%\) on CIFAR-10 and \(82.43\pm0.80\%\) on CIFAR-100, with each search completing in approximately 43.5 minutes on a single GPU. Together, these results show that short, balanced, data-dependent updates enable competitive neural architecture search across datasets and search spaces within minutes.

Thu 1 OctNeural and Evolutionary ComputingMachine Learning
The gist
Finding the best design for a neural network can be slow and costly because it requires testing many options carefully. The authors created a method called LESS that quickly narrows down good designs by using a brief training step and smart search techniques. This method evaluates candidate networks efficiently, achieving high accuracy on standard image recognition tasks much faster than comparable methods. LESS also works well without extra tuning on different datasets and larger search spaces, showing its versatility.
Open → 2610.01468v1

Gpu-initiated communication dissected for faster network operations

GPU-Initiated Communication: Dissecting Down to the Bone

Abstract: GPU-initiated communication lets GPU threads post RDMA operations directly to the NIC. It underpins NVSHMEM, NCCL GIN, and DeepEP, which serve the fine-grained, latency-critical communication of Mixture-of-Experts (MoE) models, yet its performance characteristics and optimizations remain scarcely documented beyond source code, and library comparisons fail to separate the costs of the hardware mechanism from those of the library around it. This paper dissects GPU-initiated communication at the GPU-NIC boundary. We first detail the GPU-side network path: queue placement, work-request construction, doorbell ordering, and completion semantics. We then introduce mini-gda and mini-proxy, minimal transports for the GPU and CPU-proxy submission paths, and measure them alongside NVSHMEM IBGDA, NCCL GIN, DeepEP, UCCL-EP, MSCCL++, and fabric-lib on NVIDIA H100, H200, B200, and GB200 platforms. A minimal GPU path issues an operation in 0.7 $μ$s and completes in 4.0 $μ$s; libraries add up to 4.6 $μ$s of issue time through queue management, memory ordering, and completion scope, and issue time scales with the SM clock. A tuned CPU proxy matches or beats the GPU path at idle, at the cost of a dedicated core whose operating state sets its latency and throughput. On either path, sharing a queue with bulk traffic raises latency by one to three orders of magnitude. Reaching the 260 M msg/s ceiling of our InfiniBand platform requires doorbell batching and queue parallelism, and both have resource costs: communication code can reduce GPU block residency even when unused, and all-to-all traffic loses 59% of its NIC message rate at about 3,000 active connections. The submission path alone therefore does not predict communication performance. Our experiment code and results are available at https://github.com/ParCoreLab/Dissecting-GPU-Communication-Experiments.

Thu 1 OctDistributed, Parallel, and Cluster ComputingNetworking and Internet ArchitecturePerformance
The gist
Sending data directly from a graphics card (GPU) to a network device can speed up communication in AI models but is not well understood. The authors studied how this communication happens at the hardware level and measured how quickly it operates on various GPUs and networks. They found that minimal communication can be very fast, but the surrounding software adds delays. The study also showed that sharing resources or many connections can slow things down a lot, so performance depends on more than just the GPU itself.
Open → 2610.01380v1

Learning rate scaling shows stability across hybrid transformer state models

Learning Rate Transfer for Hybrid Transformer-SSM Architectures

Abstract: We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models. In particular, we focus on the gap between the theoretical scaling rules derived for SSMs under zero-order-hold (ZOH) discretization at infinite width with growing state size, and the field-standard practical implementations using simplified-ZOH Mamba at fixed state size. Surprisingly, in this practical regime hybrid architectures achieve a near-zero LR transfer gap across widths 256-2048 and depths 4-32 up to billion-parameter scale using only the original $μ$P prescription, even though SSM operations fall outside its Tensor Programs representability conditions and every parameterization we test fails the standard coordinate-check diagnostic of $μ$P correctness. We attribute this to a two-condition decomposition of LR transfer in hybrid architectures: a global update-to-weight invariance, enforced by $μ$P's initialization and LR scaling; and a local per-component balance, provided by AdamW's per-parameter normalization. Our observations show that the optimal LR is invariant to width up to 8$\times$, that this width invariance holds across depth, sequence length, batch size, and Transformer-to-SSM ratio, and that it transfers to Nemotron-H, a production hybrid outside our custom architecture set. We hope these findings fill the gap between theoretical scaling rules and practical hybrid implementations, and stimulate further research toward bridging it.

Thu 1 OctMachine Learning
The gist
Training complex language models that mix Transformers and State-Space Models (SSMs) requires choosing the right learning rate to update the model efficiently. The authors studied existing theories and real-world practices, finding that practical models behave more predictably than expected when adjusting learning rates across different model sizes and depths. They explain this consistency by how the learning rate updates relate globally to weights and locally to individual parts, influenced by both initialization and the optimizer used. Their work helps connect theoretical rules with how models are actually trained in production.
Open → 2610.01172v1

Moe megakernels speed up gpu workloads with smart expert replication

MegaFlux: Skew-Resilient MoE Megakernels via Pipelined Expert Replication

Abstract: Mixture-of-experts (MoE) megakernels fuse expert-parallel communication with expert computation. However, under fixed expert placement, routing skew creates GPU stragglers: overloaded GPUs determine layer latency while others sit idle. Replicating hot experts can shift work to underloaded GPUs, but dynamic replicas introduce additional work: replicas must receive expert weights to execute and, during training, their partial weight gradients must be reduced at the expert owners. We present MegaFlux, which makes expert replication a runtime decision and pipelines the communication induced by replication within persistent MoE execution. An on-device planner jointly selects replica locations and assigns tile-aligned token blocks under a per-GPU replica budget, leaving router outputs unchanged. The forward and backward megakernels realize pipelined expert replication: replicas begin computation as their required weights arrive, while backward overlaps replica-gradient reduction with ongoing expert computation. MegaFlux extends TensorRT-LLM's CuTeDSL MegaMoE forward kernel and introduces a new backward MoE megakernel. Across 147 configurations per direction on eight NVIDIA B200 GPUs, MegaFlux achieves geometric-mean speedups of $1.45\times$ for forward and $1.28\times$ for backward over the same megakernels with fixed placement, peaking at $2.14\times$ and $2.64\times$. In ablations, pipelining hides $56$--$76$% of replica-weight transfer cost in forward and $91$--$100$% of combined weight-transfer and replica-gradient-reduction cost in backward, yielding up to $13.2$% and $26.7$% additional layer-latency reductions over the same replication plans with these operations executed separately. Integrated into vLLM for DeepSeek-V4-Pro prefill, MegaFlux delivers $1.13$--$1.26\times$ median end-to-end speedups over fixed placement.

Wed 30 SeptDistributed, Parallel, and Cluster Computing
The gist
Modern AI models use many 'experts' that handle parts of the work, but sometimes some GPUs get overloaded while others sit idle, slowing things down. The authors developed MegaFlux, which smartly copies busy experts to less busy GPUs and overlaps the extra data transfers with ongoing work. This pipelined approach reduces waiting times and speeds up both forward and backward passes in training. They tested MegaFlux on Nvidia GPUs and found it consistently makes large AI workloads run faster than fixed setups.
Open → 2610.00671v1

How stochastic gradient descent picks particular solutions for identity tasks

Learning the identity: a case study of how SGD selects among functional decompositions

Abstract: One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the network need only drive its weights to zero. However, this zero-weight solution is just one point on an entire manifold of population-loss minimizers, each corresponding to a different decomposition of the identity across the network's layers. Although the population loss does not distinguish among these solutions, stochastic gradient descent (SGD) reproducibly favors particular ones. For instance, under anisotropic label noise, the learned layers exhibit a noise-dependent spectrum; even with weight decay, SGD does not generally recover the zero-weight solution. Changing only the parametrization, while leaving the set of realizable functions unchanged, yields different behavior: factoring each weight matrix as a product of two matrices causes the weights to collapse to zero, even without explicit weight decay. While perhaps mysterious and unintuitive at first, these phenomena can be understood through the lens of entropic loss, which augments the population loss with a term proportional to the expected squared norm of the minibatch gradient (Ziyin et al., 2025). On the identity manifold, the population loss is constant, while the entropic term distinguishes among these decompositions. We characterize its minimizers analytically and use them to derive predictions for the structure of solutions favored by SGD. Networks trained with SGD closely match these predictions. Overall, the identity learning task studied here serves as a clean and simple case study of how the lens of entropic loss can clarify why SGD favors particular decompositions of the same input-output function.

Wed 30 SeptMachine Learning
The gist
The authors study how a common training method for neural networks, called stochastic gradient descent (SGD), chooses specific solutions when learning the identity function, which just returns its input. While there are many ways a network can represent this function, SGD tends to prefer some over others. The paper shows that this preference depends on the noise in training data and the way the network’s parameters are structured. They explain these behaviors using a concept called entropic loss, which adds a term favoring simpler changes during training. This helps clarify why SGD does not pick just any solution but a specific type among many.
Open → 2610.00615v1

Backward state choice affects learning outcomes in low precision training

Backward-State Policy Is Part of the Learning Algorithm

Abstract: Low-precision training rounds tensors that the backward pass reads again, often for several gradients; each use can read the forward's rounded value, the original, or a new random rounding. This backward-state policy looks like a memory and precision detail, settled by copy accuracy and final loss. We argue that it is part of the learning algorithm, and that neither check shows whether it is right. Copy accuracy does not decide the outcome: in three pairs of 390M runs with an emulated FP8 backward, training fails when attention's backward reuses the forward's rounded output and succeeds with a new rounding from the same distribution. Even the most accurate copy, the original itself, can be wrong by our reference: the gradient of the forward pass as it actually ran, with gradients passed through rounding unchanged. For example, a normalization output stored in low precision feeds two gradients: the gain's gradient needs the original, but the next layer's weight gradient needs the rounded value that layer multiplied. Final loss, the other check, does not rule out the error of reading the original for both: it persists in models trained with such a store, while planned loss comparisons stay within a margin fixed in advance. We therefore derive from this reference which value each use must read, or which substitute gives the same gradient on average with the forward held fixed, and check these per-use requirements on single operators, without training. In three tests using PyTorch and Transformer Engine, the requirements predicted beforehand whether reuse changes what the backward computes on average relative to an independent copy, and every prediction held. Backward-state policy is thus part of the learning algorithm: it should be specified and checked use by use, not settled by copy accuracy and final loss.

Wed 30 SeptMachine Learning
The gist
Training AI models often uses shortcuts when copying numbers during calculations, especially in lower-precision formats to save space and time. The authors found that how these copied numbers are reused in backpropagation, specifically whether the backward step uses the same rounded number or a fresh one, actually changes the learning result. This means the choice of which value to use in each step is not just a technical detail but part of how the model learns. They proposed a way to decide which value to use for each calculation, making these choices explicit and testable without full retraining.
Open → 2609.39813v1

Zero order training cuts memory use for large AI models fast

Probe-Space Preconditioning for Fast and Stable Zero-Order Training

Abstract: Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU memory (assuming batch size 8 and sequence length 2048). Alternatively, zero-order optimization (ZOO) trains in inference-mode (requiring only $\approx$ 60GB for the same model): no stored activations, no gradients, and no optimizer states. However, ZOO convergence has lagged behind BP. In this work, we evaluate two methods to close this gap. First, we show that reallocating training compute budget from many steps to large effective batch sizes with many perturbations (or probes) but fewer steps, allows 1SPSA (Spall, 1992) to outperform zero order methods like MeZO (Malladi et al., 2023) with less training compute. Next, we introduce 1.5-SPSA, adding a single "clean" forward-pass per step to 1SPSA to calculate a cheap diagonal preconditioner in probe-space, which improves convergence rate and convergence by down-weighting high curvature directions. Benchmarking on 6 post-training datasets on both Qwen3 and OPT model families, we show that 1.5-SPSA achieves State-of-the-Art results over previous ZOO solvers with much less optimization steps. For example, we train OPT-13B (for direct comparison to MeZO) and find 1.5-SPSA achieves +3.1% accuracy on SST-2 over both MeZO and BP in only 70 steps vs. MeZO's 100,000 steps. Finally, we combine an 8-bit-packing random generator, triton fused unpack/apply kernels, and distributed parallelism to achieve fast and stable training of models as large as OPT-30B in-place on commodity GPUs (e.g. A100).

Tue 29 SeptMachine Learning
The gist
Training very large AI models usually requires a lot of computer memory for storing extra information, which can be a problem. The paper looks at a different kind of training called zero-order optimization that needs much less memory but has been slower. The authors improve zero-order training by changing how many trial guesses are made each step and adding a simple extra check that helps the training focus on important directions. Their new method trains big models more quickly and efficiently than before, using ordinary GPUs.
Open → 2609.38095v1

Dynamic width expansion helps neural networks learn new classes better

Width Expansion as a Method for Class Incremental Learning

Abstract: Class Incremental Learning (Class-IL) requires models to learn new classes over time while preserving previously acquired knowledge without access to past data or task identity. This setting intensifies the stability-plasticity dilemma and makes catastrophic forgetting a central challenge. Existing approaches include regularization, knowledge distillation, replay, and architectural expansion. However, many expansion methods rely on explicit task identifiers or predefined growth strategies, limiting their applicability when task boundaries are unavailable at inference time. This work proposes a dynamic width expansion method that increases the number of neurons within existing layers according to a normalized loss criterion, without requiring task-specific information. An attention mechanism with persistent key-value memory is also incorporated to stabilize feature representations and reduce interference between previously learned and newly introduced classes. The approach is evaluated on Split MNIST and Split CIFAR-100 under the standard Class-IL protocol. Experiments compare fixed-capacity and dynamically expanding architectures, both with and without attention, combined with established continual learning methods including EWC, LwF, and A-GEM. Results show that progressive width expansion consistently improves performance over fixed architectures, particularly when combined with functional methods and A-GEM. The combination of width expansion and attention provides the most consistent gains. Overall, dynamic width expansion based on representational demand provides an effective and flexible strategy for Class-IL, although uncontrolled growth may increase overfitting and computational cost.

Tue 29 SeptMachine LearningArtificial Intelligence
The gist
When computers learn new categories of things over time, they often forget what they learned before. The authors propose a way for neural networks to add more neurons only when they need to, which helps them remember old information while learning new stuff. They also use a special memory-attention system to keep the new and old knowledge from mixing up too much. This approach was tested on image tasks and worked better than keeping the network size fixed.
Open → 2609.37702v1

LionMuon optimizer speeds up language model training with less compute

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

Abstract: Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every $P$ iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per $P$ steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with $P=2$ and $P=5$ reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon

Mon 28 SeptMachine Learning
The gist
Training large language models requires a lot of computer power, and the choice of optimizer can save significant resources. The authors introduce LionMuon, a new optimizer that mixes two types of update steps to balance speed and accuracy. It runs an expensive but strong step occasionally and cheaper steps in between, reducing overall computation and communication costs. Tests show LionMuon reaches better training results faster than several popular optimizers.
Open → 2609.35297v1

Torch-PIM speeds up deep learning by smartly moving work to memory cores

Torch-PIM: Automated Profile-Guided PIM Offloading for PyTorch

Abstract: Modern deep learning (DL) workloads are limited by data movement, and processing-in-memory (PIM) targets this bottleneck by placing compute units near the memory. However, PyTorch and other DL frameworks lack compiler support for making this decision on the code they lower: existing offloading frameworks target hand-written C/C++ programs, while those that address DL fix the candidate set to a list of operator types before lowering. We present Torch-PIM, a compiler framework that uses profile-guided optimization (PGO) to decide host-versus-PIM placement over the loop nests that progressive lowering materializes. Every parallel loop nest the pipeline emits enters the candidate space, and each is assessed in two stages: the amount of work it carries, and its memory boundedness. Every quantity the assessment consumes is profiled on the host or obtained from the multi-level intermediate representation (MLIR) of the code. Across PIM configurations of 32 to 128 cores, Torch-PIM's offloading decisions yield speedups of up to 8.6x on tensor operators, 2.9x on MLP, 4.4x on Attention, 5.1x on GPT-J-6B, and 3.6x on LLaMA-7B over CPU-only execution.

Mon 28 SeptHardware ArchitectureDistributed, Parallel, and Cluster Computing
The gist
Deep learning often slows down because data has to move a lot between memory and processors. The authors created Torch-PIM, a tool that automatically decides which parts of a deep learning program should run close to memory to save time. It looks carefully at the program’s loops and uses information from running the program to make these choices. Using Torch-PIM can make common deep learning tasks run several times faster compared to only using the main processor.
Open → 2609.34657v1

Local learning methods face limits compared to backpropagation in deep networks

The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

Abstract: The Forward-Forward Algorithm (FFA) replaces backpropagation (BP) with layer-wise local contrastive objectives, eliminating the backward pass and the need to retain intermediate activations, yet suffers a persistent performance gap with BP that worsens with depth. This paper diagnoses two structural failure modes: an optimization floor arising from concurrent local updates; and a geometric collapse of layer representations driven by the local update mechanism. On the optimization side, we prove that the FFA loss satisfies the Polyak--Lojasiewicz inequality at each layer; however, simultaneous layer updates induce inter-layer representation-distribution drift, so each layer optimizes against a moving input distribution and incurs an error floor. On the representational side, the pairwise similarity kernel of layer representations contracts exponentially toward rank one as depth increases, collapsing the diversity of per-layer error signals. This collapse bounds FFA's effective learning capacity, which measures the diversity of gradient information across layers, independently of depth, whereas BP's chain-rule signal preserves per-layer diversity, yielding a capacity that scales with depth.

Sun 27 SeptMachine Learning
The gist
Backpropagation is a common way to train deep learning models, but it requires information flowing backward through the network. The Forward-Forward Algorithm tries to avoid this by training layers separately using only local information, which simplifies training. The authors found that this method struggles because layers affect each other unpredictably and their internal representations lose diversity as the network gets deeper. This limits how well the model can learn compared to standard backpropagation, especially in deeper networks.
Open → 2609.33240v1

Muon orthogonalization limits in noisy gradient optimization near optima

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Abstract: Muon replaces the momentum buffer of each weight matrix by its orthogonal polar factor. We ask what this orthogonalization does near an optimum, where minibatch noise dominates the gradient. Under a Gaussian noise model, the expected Muon update becomes a scaled gradient step, so a linear method with a suitably matched learning rate reproduces Muon's first-order mean response. The stochastic update is a different matter: after the response is matched, Muon retains a nonlinear residual that is uncorrelated with the input noise and contributes additional covariance. A Hermite expansion shows how momentum acts on this residual. Its higher-order components decorrelate faster than the linear component, so momentum suppresses the residual's accumulated covariance relative to the linear part, but never eliminates it. In a local quadratic surrogate that evaluates the residual on the stationary noise buffer, the residual adds stationary covariance and raises the stationary loss floor at every stable step size, while leaving the contraction dynamics unchanged. Simulations of the full nonlinear recursion on quadratics and measurements on frozen transformer gradients support each step of this picture. Together, the results make a theoretical case for replacing orthogonalization by response-matched momentum SGD once optimization becomes noise-dominated.

Sat 26 SeptMachine Learning
The gist
Training deep learning models involves updating many numbers called weights using gradients estimated from small data batches, which can be noisy. The paper studies a method called Muon that keeps some parts of these weights orthogonal to stabilize training. The authors find that near an optimal solution where noise is significant, Muon's orthogonalization behaves like a scaled version of simpler momentum methods but also introduces extra complicated noise effects. These extra effects never fully go away and raise a baseline level of loss during training. The work suggests simpler momentum methods might be better when noise is dominant during optimization.
Open → 2609.32861v1

LocalProp cuts memory use in deep learning by updating weights locally

LocalProp: Neuro-Localized Memory-Efficient Backpropagation

Abstract: The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor memory efficient, since learning inside the brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a pruning operation is performed, followed by a short final fine-tuning phase. Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several datasets, including large-scale benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-memory trade-off.

Sat 26 SeptArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
The gist
Training deep learning models usually involves adjusting all the parts of the model at once, which uses a lot of computer memory and is not how brains learn. The authors propose LocalProp, a method where only small sections of the model are updated at a time, making training more memory efficient. They use a step called pruning to pass learning signals from higher to lower parts of the model and finish with a final fine-tuning. Experiments show LocalProp performs well on big image datasets while using much less GPU memory than traditional methods.
Open → 2609.32517v1

Direct feedback alignment reveals common error collapse slows learning

Common-Mode Collapse and Recovery in Direct Feedback Alignment

Abstract: Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivity across 48 settings. On MNIST, class decodability largely survives collapse, but readout learning remains slow at a fixed learning rate. Adam learns faster despite deeper collapse. Calibrating the baseline readout to the class prior suppresses collapse and speeds learning; weaker feedback trades less collapse for slower learning. Replacing errors by their signs sustains collapse; subtracting the signal's batch mean prevents sustained collapse and improves learning in the tested setting. Related effects occur in deeper and convolutional networks and on CIFAR-10, with severity and cost depending on the readout, optimizer and input statistics.

Fri 25 SeptMachine LearningNeural and Evolutionary Computing
The gist
Training certain neural networks can get stuck because they share a common error pattern that makes hidden units saturate and stop learning effectively. The authors found that this happens when the network’s hidden units respond too similarly, a problem they call 'common-mode collapse.' They show that adjusting the way the output error is fed back and how learning rates are set can reduce this problem and speed up training. Their study tested different settings, networks, and datasets to understand when and why this collapse happens.
Open → 2609.31589v1

Neural network weights optimized to use smaller variable grammar codes

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

Abstract: We show that neural network weights can be explicilty fintuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a straight-through estimator. The int8 weights of the network are flattened into one string, and near-matching Re-Pair patterns are made exactly equal within a global L2 budget. The network computes with the rewritten weights and trains through them with a straight-through estimator. Unlike a flat codebook of fixed-size entries, a grammar offers variable-length patterns and reuses them hierarchically inside larger ones. On the MLP weights of ViT-B/16 and ViT-L/16 finetuned on CIFAR-10, WeightPE produces a Re-Pair grammar 0.43x and 0.38x the size of the one produced by an equivalent int8 QAT run, at a cost of 1.9 and 1.1 accuracy points. The trend extends to different grammar compressors (LZ78, SEQUITUR), over which the networks has not be finetuned against. To our knowledge, this is the first time grammar size has been used as an explicit training objective for network weights.

Fri 25 SeptMachine Learning
The gist
Deep learning models use many numbers called weights, which can take up a lot of space. The authors show a way to tweak these weights so they can be described by shorter, repeating patterns called grammars, saving memory. They do this by grouping similar weight values together and training the model to work well with those simplified weights. This makes the weight representation smaller but with only a small drop in accuracy.
Open → 2609.31564v1

Dynamic tensor memory policies show sudden slowdowns and failures

Deterministic Regime Switching and Feasibility Inversion in Dynamic Tensor Rematerialization

Abstract: We report fine-grained, deterministic instability in Dynamic Tensor Rematerialization (DTR), an online eviction policy for memory-constrained DNN training, measured on the reference DTR simulator (simrd) using public execution traces. On an LSTM trace, memory budgets differing by 0.10% of unconstrained peak memory select fast and slow execution regimes whose overheads differ by as much as 7.3x; the slow regime is driven by broadly repeated re-eviction of the same storages (evictions per storage rise from 1.33 to 8.27 while the set of distinct evicted storages is essentially unchanged: 5,233 vs 5,236, with the two sets overlapping at Jaccard 0.999). On a ResNet-32 trace, a fine budget sweep reveals a deterministic feasibility inversion: the run is feasible at ratio 0.101, infeasible (OOM) across 0.102-0.106, and feasible again from 0.107. We trace the immediate cause of the OOM to a fully pinned recursive rematerialization frontier that exceeds the budget after every evictable tensor has been evicted. Ablations using the DTR authors' own variants implicate the joint size-staleness scoring term in the observed LSTM instability. We argue these are at least two distinct budget-sensitive pathologies rather than one mechanism, and we separate what is demonstrated from what remains hypothesised. All results concern the reference simulator; reproduction in a production runtime is future work. Code, instrumentation, and raw results accompany this preprint.

Fri 25 SeptMachine LearningDistributed, Parallel, and Cluster Computing
The gist
Training deep neural networks needs careful memory management. The authors studied a memory-saving method called Dynamic Tensor Rematerialization and found it can suddenly slow down a lot or even run out of memory depending on tiny changes in the memory budget. They saw these effects using a computer simulator on two types of networks, LSTM and ResNet-32. The paper explains some causes of these sudden changes but notes that these findings are based on simulations, not yet tested in real systems.
Open → 2609.31250v1

DanLing NestedTensor speeds up deep learning with variable-size inputs

DanLing NestedTensor: Composable Multi-Ragged Tensors for Deep Learning

Abstract: Variable-size inputs are common in deep learning, but dense batching allocates a shared envelope and spends computation on padding. The cost multiplies across varying axes: an explicit pair state allocates $BN_{\max}^2$ positions instead of $\sum_i N_i^2$. Packing removes that waste, but composing packed operations still requires the logical axes and sample boundaries a flat buffer no longer exposes. We present DanLing NestedTensor, a PyTorch tensor abstraction that makes multi-ragged structure a property of the tensor itself. Packed values carry tensor-backed partitions and logical dimension order, so broadcasting creates ragged axes, feature transformations retain them, and reductions consume them. The same representation carries through autograd and both eager and compiled execution. On an A100, the geometric-mean speedup over same-mode padding is 2.74$\times$ eager and 3.39$\times$ compiled across four BERT scales, and 1.97$\times$ eager across four FCN backbones. A four-block Pairformer-style workload runs 2.40-4.32$\times$ faster than a padded reference using native PyTorch kernels across square length regimes in eager execution, with peak allocation falling from 38.08 to 5.41 GiB on its high-variation batch. The tensor interface lets model code built from its supported operators compose efficient variable-size computation without managing offsets at any call site. Code will be released publicly upon publication.

Thu 24 SeptMachine LearningArtificial IntelligencePerformance
The gist
Handling inputs of different sizes in deep learning often wastes computing power when everything is forced into fixed-size formats with padding. The authors designed DanLing NestedTensor, a new way to represent data that keeps track of varying sizes naturally inside the tensor itself. This approach speeds up computation and reduces memory use by avoiding padding and managing complex data layouts more efficiently. It works transparently with common deep learning tools like PyTorch and maintains support for training and inference.
Open → 2609.30379v1

KernelOPT improves GPU kernel speed by optimizing compiler outputs

KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel Optimization

Abstract: Deep learning inference and training performance depends critically on GPU kernel efficiency. Modern compilers such as PyTorch Inductor automatically generate GPU kernels from high-level model code, but frequently underperform expert-written implementations by wide margins. Recent LLM-assisted kernel optimizers can close this gap for standalone kernels, yet treat compiled models as black boxes, generally optimizing individual standalone kernels without respecting the compiler's structural decisions or verifying the model end-to-end. We present KernelOPT, a multi-agent system that treats compiled models as structured artifacts. It preserves vendor library calls (cuBLAS, cuDNN) and exclusively targets generated Triton sub-kernels using five profiling-guided LLM agents. A four-gate verification cascade of static validation, multi-seed correctness, model-level float64-fallback verification, and performance gating filters candidates during optimization and verifies the re-stitched model end-to-end. If no candidate passes all four gates, the system preserves the compiler baseline. The system accepts PyTorch nn.Modules, standalone Triton kernels, and Helion kernels. Evaluated on 250 KernelBench problems, KernelOPT achieves geometric mean speedups over \texttt{torch.compile} of 1.40$\times$ (Level 1: 51/100), 1.15$\times$ (Level 2: 31/100), and 1.07$\times$ (Level 3: 12/50) across all problems.

Thu 24 SeptDistributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
The gist
Deep learning run times strongly depend on how efficiently GPU code is written, but automatically generated GPU programs are often slower than expert ones. The authors present KernelOPT, a new tool that safely improves GPU code by focusing on specific parts while keeping trusted library calls intact. It uses multiple AI helpers to propose improvements and checks that the overall program still works well before adopting changes. Tests on many problems show KernelOPT can make GPU code run up to 40% faster than current compiler outputs.
Open → 2609.30059v1

Error supervising neural network detects CNN parameter faults

ESupNNet: An Error Supervising Neural Network architecture for error detection against soft errors in parameters

Abstract: This work presents a novel approach to detect misclassification errors in CNNs caused by soft errors in their parameters. We propose an architecture that uses inter-class relations induced by the CNN that needs protection. The architecture has minimal resources overhead and does not require modifying the CNN, which makes it a competent solution that can be used with other error protection techniques. We have validated the architecture with five different combinations of modern dataset-model pairs: ImageNet-1K for ResNet-50 and EfficientNetV2-Small; CIFAR-10 for MobileNetV3, ShuffleNetV2-Small with 2.0x output channels and MNASNet with depth multiplier of 1.3. The validation process was done rigorously with statistical significance, from the creation of the datasets used by the architecture to the acquisition of experimental results. Results show great performance with over 90% accuracy in detecting single errors and great error detection over multiple Bit Error Rates, which can be potentially increased with hyperparameter tuning.

Tue 22 SeptHardware Architecture
The gist
Deep learning models like CNNs can sometimes make mistakes because of tiny errors in their settings caused by things like hardware glitches. The authors created a small extra neural network that watches the main CNN’s behavior to catch when these errors might cause a wrong answer. This helper network uses relationships between categories the CNN knows to spot problems without changing the original model. They tested this system on several popular image recognition models and datasets and found it detects errors with over 90% accuracy.
Open → 2609.26374v1

Neural spectral capacity predicts and optimizes model design efficiently

Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Abstract: Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko-Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU -- a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT, $τ= 0.505$ on pairs differing in #Params by less than 10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, about 5900x faster than the strongest training-free proxy baseline.

Sat 19 SeptMachine LearningArtificial Intelligence
The gist
Choosing the right structure for AI models often involves balancing size and computing power, but common measurements miss key details about design. The authors introduce Neural Spectral Capacity (NSC), a quick way to predict a model's potential by looking at its weight matrices without needing to train it or use data. They also provide NSC-DP, a method that quickly finds the best architecture layout under limits like size or compute. Their method ranks models more accurately than traditional measures and speeds up tasks like pruning large models without extra data.
Open → 2609.23087v1

Unified 16-bit format boosts neural network error protection efficiency

UniCASE: A Unified 16-bit Floating-Point Format with Criticality-Aware Selective ECC for Efficient DNN Protection

Abstract: Soft errors are an increasing reliability concern for Deep Neural Network execution because they can corrupt parameters, leading to accuracy degradation. While conventional ECC offers strong fault protection, it incurs additional parity storage and computational overhead. Embedded-parity formats reduce storage cost by reusing the least-significant bits, but they do not optimize protection while reducing computational overhead. We propose UniCASE, a unified 16-bit floating-point (FP) format that jointly optimizes data representation and error protection for reliable DNN execution. It identifies stable blocks across FP64, FP32, FP16, and BFloat16 that can be mapped into a unified representation. Based on bit-level criticality analysis, UniCASE uses selective ECC that assigns distinct levels of protection to different data bits according to their resilience against soft errors. Experimental results show that UniCASE reduces encoder/decoder cost by up to 30%, preserves model accuracy within 1% of the FP32 baseline, and provides significantly stronger soft error resilience than existing embedded-parity methods.

Fri 18 SeptHardware ArchitectureCryptography and Security
The gist
Soft errors can cause mistakes in how deep learning models work, which harms their accuracy. The authors propose UniCASE, a new way to store numbers that both saves space and protects against these errors more efficiently. Their method looks at which parts of the number are most important and protects them better, reducing the overhead of error correction. Tests show UniCASE keeps accuracy close to standard formats while cutting error correction costs and improving reliability.
Open → 2609.22590v1

Matrix adaptive optimization improves neural network training stability

Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methods

Abstract: Adaptive optimization methods such as AdaGrad and Adam are widely used in modern neural-network training, but their adaptive scaling is primarily designed for vector-valued parameters and does not explicitly exploit matrix structure. Recent matrix-aware optimizers demonstrate the benefits of structured optimization, yet a general theoretical framework for deriving matrix-aware adaptivity comparable to that of AdaGrad remains lacking. In this work, we develop a general Online Mirror Descent framework with adaptive proximal functions for matrix-valued parameters, providing a principled approach to deriving matrix-aware adaptive optimization through online regret minimization. By introducing row-wise and column-wise matrix proximal functions and analyzing the resulting regret trade-off, we derive Row-wise Matrix AdaGrad (Row-AdaGrad) and Column-wise Matrix AdaGrad (Column-AdaGrad), with adaptive scaling determined by the accumulated row-wise or column-wise gradient norms. We establish regret guarantees and show that these matrix-aware bounds can be strictly tighter than those of entry-wise AdaGrad under structured gradients. Experiments on matrix factorization and deep neural-network training further demonstrate the benefits of aligning adaptive scaling with matrix structure, including improved optimization stability and trainability at larger learning rates and greater network depths.

Fri 18 SeptMachine LearningArtificial Intelligence
The gist
Optimizing neural networks often involves adjusting many parameters arranged as matrices. Typical methods adapt learning rates for each parameter individually, missing patterns in the matrix structure. The authors develop a new approach that adapts learning rates by looking at entire rows or columns of these matrices, making training more stable and effective. Their method shows better performance when parameters have structured gradients and helps train deeper or larger neural networks.
Open → 2609.21815v1

Tensor program bugs found faster by checking outputs one location at a time

The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs

Abstract: Tensor programs, as used in deep learning models, are a prime target for optimization, as small performance improvements can have a large impact across training or inference workloads. However, such optimizations are complicated and can produce subtle bugs. Traditionally, correctness is assumed when differential testing against a reference on random inputs fails to reveal bugs. However, the inputs to these programs are massive tensors, and finding bugs can require generating extremely low likelihood inputs with precise relationships among their values. We propose a novel way to find bugs more consistently by flipping the quantifiers. Rather than generating a single input and checking all output tensor locations for equivalence, what if you could check a single output tensor location's equivalence for all inputs? We implement this idea in a system, \dirigo, by using a novel symbolic execution strategy. We demonstrate that \dirigo can find bugs effectively in a public dataset of 6,988 AI-written CUDA kernels that are all marked correct by differential testing. Of these, \dirigo finds 600 kernels that are actually buggy, and finds 97.3\% of those bugs within two minutes.

Thu 17 SeptProgramming LanguagesMachine LearningSoftware Engineering
The gist
Tensor programs used in AI models are tricky to optimize because small bugs can cause big problems. Traditional tests check many outputs for one input, but this misses rare bugs hidden in complex data. The authors flipped this by checking one output location for all possible inputs, catching subtle mistakes more reliably. Their system, Dirigo, found hundreds of bugs that older tests missed, most within minutes.
Open → 2609.19611v1

Sharding technique speeds up transformer retraining and distillation

Affinity-Aware Sharding for Delayed Tensor Parallelism

Abstract: Delayed Tensor Parallelism (DTP) removes the blocking all-reduce of tensor-parallel Transformer inference. Every device adds its own partial output to its residual stream (and broadcasts it) immediately, but only gathers (receives) the other devices' partials $δ$ modules later. A TP to DTP change therefore amounts to a real architecture change, and dense Transformer models need to be retrained or distilled after adaptation. We show that DTP breaks the permutation symmetry of neurons inside FFNs and of KV heads inside attention modules, and that this symmetry breakage makes the sharding itself a modelling decision. We show that maximising the affinity between the KV heads and the FFN neurons co-located on a device, by permuting the dense model before sharding, speeds up the distillation or retraining process. The affinity is measured with a first-order approximation of the damage that losing a head's contribution does to each neuron's output, and the co-located affinity is maximised with a coordinate-ascent optimiser that alternates an exact balanced assignment of neurons with an exhaustive search over the KV head partitions. The whole procedure takes under two minutes on one GPU for Qwen3-0.6B and Danube3-500M. On these models, at $δ=1$, the affinity-optimised layouts reach any distillation target in about half to two thirds of the steps needed by the naive contiguous layouts, over the whole 10k-step range we tested, and every optimised seed beats every contiguous seed and all but one of the sixteen random layouts. We also show that the co-located affinity score at initialisation predicts the KL to the base model after training, across seventeen layouts ranging from anti-optimised to optimised (Pearson $-0.81$ and $-0.89$).

Sat 12 SeptMachine LearningComputation and LanguageDistributed, Parallel, and Cluster Computing
The gist
Transformer models are large AI programs that can be split across many devices to work faster. One way to do this, called Delayed Tensor Parallelism (DTP), changes how parts of the model communicate, which requires retraining the model after splitting it differently. The authors found that rearranging the parts shared on each device to match how they interact speeds up this retraining. They show a method to find the best arrangement of these parts, making the retraining process about twice as fast on tested models.
Open → 2609.13846v1

Lightweight method to check GPU code errors in deep learning training

PEAT: Pseudo-Error Assessment for GPU Kernel Validation in DNN Training

Abstract: Deep neural networks (DNNs) are widely adopted in various fields, driving an emerging trend in developing software stacks associated with DNN training systems. For example, many codes have been ported across different frameworks or developed to leverage the computing power of GPUs or domain-specific accelerators. However, validating a kernel implementation in DNN training is time-consuming and generally requires massive storage. Specifically, this poses a fundamental question: how to characterize the behavior of a new implementation when it is integrated into a DNN training flow. Unfortunately, this problem is not well investigated in the literature, to the best of our knowledge. To address this shortcoming, we present PEAT - a lightweight inspection framework for \underline{P}seudo-\underline{E}rror \underline{A}ssessment associated with GPU kernel validation in DNN \underline{T}raining. Firstly, inspired by conventional fault injection (FI), PEAT's Profiler invokes an operation-wise kernel in a training flow to collect a DNN model's states (e.g., checkpoints and activations). More importantly, the Profiler introduces two simple yet effective techniques, playback FI and frequency-based runtime FI, leveraging persistent kernel calling during the training process. Secondly, PEAT's Analyzer characterizes profiled errors, revealing some signatures from the error distribution of a kernel compared to the golden one. Lastly, PEAT's Detector provides some guidelines as a sufficient condition, which enables associating several well-known error models with signature patterns. We demonstrate the applicability of our approach by presenting the results and analysis using GPUs from the two most popular vendors, NVIDIA V100 and AMD MI250, on various AI models, from vision tasks to language models, for both pretraining and finetuning scenarios.

Fri 11 SeptDistributed, Parallel, and Cluster ComputingComputational Engineering, Finance, and ScienceSoftware Engineering
The gist
Running deep learning programs on GPUs needs special code called kernels, and making sure these kernels work correctly is hard and takes a lot of time and storage. The authors designed a tool named PEAT that watches GPU kernels closely during training, collects information about how the model's data changes, and looks for patterns in errors. This helps identify if the kernels produce unusual errors compared to trusted versions, which means programmers can find problems faster. They tested PEAT with GPU hardware from popular makers and on several AI tasks like image and language processing.
Open → 2609.13544v1

Learning rate schedules inspired by fastest descent improve deep network training

BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

Abstract: The learning-rate schedule is a consequential choice in training deep networks, yet the policies in common use are heuristic, and published comparisons are hard to read, because architecture, dataset, and budget tend to vary alongside the schedule. We study BrachistoneLR, a schedule built by mapping the vertical coordinate of the brachistochrone, the curve of fastest descent under gravity, onto the range between a peak and a floor rate. Expanding the definition shows it to be cosine annealing with the half-period set to E - 1 instead of E, the configuration a standard implementation gives when its period argument is one less than the number of epochs. The rate therefore reaches its floor at the last epoch trained rather than one epoch later, and we show this difference decays as E^-2, making it a short-horizon effect. We then benchmark six schedules over 72 runs on three image classification datasets (MNIST, Fashion-MNIST, CIFAR-10) and four architecture families (fully connected, convolutional, recurrent, residual), fixing the optimizer, data pipeline, and evaluation protocol so that only the schedule varies. Schedules that fall smoothly from peak to floor beat the constant rate and calendar-based decay by margins that grow with task difficulty, reaching 2.5 points of dataset mean on CIFAR-10. Within that leading group, BrachistoneLR, cosine annealing, and warmup-cosine lie within 0.06 accuracy points and 0.17 of a mean rank, which one seed per configuration cannot separate. BrachistoneLR is best on both residual networks and has the highest CIFAR-10 mean, and it sets no milestones, decay factor, warmup length, or restart period. We conclude that the shape of a schedule matters more than its parameterization, that the choice of whether to use a smooth schedule matters more than the choice among them, and that the terminal-rate distinction is worth attention only over short horizons.

Tue 8 SeptComputer Vision and Pattern Recognition
The gist
Training deep learning models needs careful adjustment of the learning rate, which affects how quickly the model learns. The authors study a new schedule called BrachistoneLR, inspired by the curve of fastest descent in physics, and show it is a slight variation of the popular cosine annealing schedule. They compare six schedules on several datasets and network types while controlling other variables. The results show that smoothly decreasing schedules outperform constant or step-based ones, and BrachistoneLR performs slightly better for some networks without needing extra tuning.
Open → 2609.08069v1

Kan training scales efficiently on multi gpu high performance computers

Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems

Abstract: Kolmogorov-Arnold Networks (KANs) replace the fixed activation functions and linear weights of Multi-Layer Perceptrons (MLPs) with learnable univariate functions on network edges, offering improved interpretability and, in some settings, competitive parameter efficiency. While the approximation properties of KANs have received considerable attention, their behavior under distributed, multi-GPU training has not been systematically characterized. This paper presents an empirical scalability study of data-parallel KAN training on multi-node, multi-GPU high-performance computing (HPC) infrastructure, evaluated along four dimensions: strong scaling, weak scaling, communication overhead, and model-size scaling. Experiments were conducted on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel (DDP). KAN training reaches 74.7% parallel efficiency at 8 GPUs with a 5.97x speedup, consistent with conventional deep learning workloads. Weak scaling shows an initial single-to-multi-GPU throughput drop followed by strong stability. Communication overhead follows a non-monotonic pattern (1.3%-6.1%), driven primarily by All-Reduce algorithm selection and inter-node latency rather than KAN's edge-wise gradient structure. The parameter-to-memory ratio improves with model size even as training time scales unfavorably. These results indicate that operator-level and data-parallel optimizations for KAN are complementary. We provide deployment guidelines for GPU topology and model-size selection, and discuss the limitations of a synthetic-regression evaluation.

Mon 7 SeptDistributed, Parallel, and Cluster ComputingMachine LearningPerformance
The gist
Kolmogorov-Arnold Networks (KANs) are a special type of neural network that use learnable functions instead of fixed math rules to make decisions. The authors studied how well KANs train when the work is split across many GPUs in a supercomputer. They found that KAN training gets faster almost as expected when adding more GPUs, with some small delays due to communication between machines. Their experiments help show how to best set up hardware and model size for training KANs efficiently.
Open → 2609.07740v1

Kolmogorov Arnold stability holds for discontinuous deep learning functions

Kolmogorov--Arnold stability for discontinuous functions

Abstract: Here we investigate the stability of the Kolmogorov--Arnold representation theorem (KART) under adversarial reparameterisations of the hidden layer for multivariate discontinuous and unbounded functions. Our results provide a rigorous mathematical foundation for the structural robustness of modern deep learning architectures, such as Kolmogorov--Arnold Networks (KANs), under adversarial configurations.

Mon 7 SeptMachine Learning
The gist
The paper studies how a mathematical way to represent complex functions, called the Kolmogorov–Arnold theorem, stays reliable even when the function is jumpy or unsteady. The authors explore how this stability works when deep learning networks face tricky changes that try to confuse or break them. Their findings support why certain deep neural networks remain robust when their inner layers are reconfigured in adversarial ways. This helps explain why these networks can perform well even under challenging conditions.
Open → 2609.07240v1