Sparse Competition during Training For the Emergence of Specialized Modules
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explore how groups of neurons in deep neural networks can naturally form modules that specialize in different parts of a task during training. They introduce a method that keeps accuracy high while routing inputs sparsely, leading to neuron groups that respond to specific input classes like dogs or vehicles. This happens without explicitly telling the network to create these modules, and the authors show this on datasets like ImageNet-100 and CIFAR-100. Their work suggests that competition among neurons can help create useful modular structures in standard neural networks.
modularitydeep neural networksneuron groupssparse routingdisentangled representationsspecializationImageNet-100CIFAR-100functional modularitycompetitive dynamics
Authors
Baptiste Rossigneux, Karim Haroun
Abstract
Modularity in deep neural networks has been proposed as a means of improving both interpretability and training by promoting disentangled representations and reducing redundancy. In this work, we study the emergence of modular structure through competition dynamics between groups of neurons during training. We introduce a method that (i) maintains near-baseline accuracy, (ii) induces usage-based modularity by sparsely routing inputs to neuron groups, and (iii) encourages specialization of these modules, such that their activations are correlated with input classes. We evaluate the proposed approach on ImageNet-100 and CIFAR-100 and show that with it, specialized modules emerge without module-level supervision. These modules capture a meaningful high-level structure in the data, with individual modules responding to semantic categories (e.g., dogs or vehicles). We also study the emergence of a hierarchical partition of sub-tasks depending on the number of modules. Our results suggest that competitive dynamics can serve as a simple mechanism for inducing functional modularity in standard architectures.