Residual networks use dynamic patterns to route information internally
Do ResNets Route? Sparse Interaction Experts in Residual Networks
Machine Learning
Summary
Residual networks are a type of deep learning model that always apply every step of their process, but this paper finds that different steps contribute differently depending on the input. The authors analyze how these networks combine various parts and show that the importance of these parts varies and interacts in complex ways. Unlike traditional routing which skips parts, these networks softly route by weighting contributions from different sections without turning any off completely. This reveals a new understanding of how these models flexibly handle diverse inputs internally.
What this means in practice
- •For deep learning engineers: Design models that adjust internal component contributions dynamically without explicit routing for more flexible inference.
- •For ai system architects: Analyze and improve large residual networks by understanding their implicit soft routing and interactions to optimize model design.
Authors
Liang Yan, Siying Chen, Kaijie Chen, Bo Li, Jinghao Zhang, Mu Miao
Abstract
Residual networks execute every block for every input, yet their functional contributions need not be input independent. We formulate a trained ResNet as a set function over binary residual-branch masks and apply Möbius inversion to decompose its output exactly into individual residual corrections and higher-order interactions. For smooth residual stacks, we show that each fixed $k$-way interaction scales as $\mathcal{O}(λ^k)$ under residual scaling. Exhaustive analysis of ImageNet-pretrained ResNet-18 and ResNet-34 reveals that interaction mass peaks at orders five and ten, respectively, rather than at low orders. Reducing the residual scale shifts both spectra toward lower orders, but also changes model predictions. The interaction coefficients are concentrated in magnitude but not hard sparse, and prediction-preserving sparsity weakens with depth. Crucially, the dominant interactions vary across inputs and predicted classes around a shared global core, while their overall complexity changes little with sample difficulty. These results show that dense ResNets implement an implicit form of soft routing: every block is executed, but different inputs rely on different residual interaction experts. Routing can therefore emerge at the level of functional contribution without an explicit router or sparse execution.