Sparsity helps neural networks develop specialized task modules
Sparsity-driven emergence of functional modularity in multitask neural networks
Machine LearningArtificial Intelligence
Summary
Neural networks can learn many tasks, but it’s unclear how they organize themselves to handle different jobs efficiently. The authors show that encouraging networks to be sparse—that is, to use fewer connections—helps them form specialized parts that focus on specific tasks. These parts, called functional tokens, appear mainly in deeper layers of the network while earlier layers stay more general. The study found that these specialized parts can be reused or combined to build new functions without retraining.
What this means in practice
- •For machine learning engineers: Design multitask neural networks that efficiently reuse specialized modules for better performance and interpretability.
- •For software architects: Implement sparsity constraints to create modular neural network components that can be recombined without retraining for flexible AI systems.
Authors
Bowen Shi, Qi Ouyang, Long Qian
Abstract
Neural networks can learn multiple tasks, but how learning organizes shared computation into specialized and reusable components remains poorly understood. In this work, we show that sparsity inducing regularization promotes functional modularity in multitask networks without predefined module boundaries. Using functional profiling in multilayer perceptrons and a lightweight transformer, we identified compact, task-selective subnetworks, termed functional tokens. Across rule-based task families, selectively retaining these tokens preserves their associated functions while reducing performance on other tasks. Such specialization develops preferentially in deeper layers during training, while earlier layers retain shared computation. Regularization strength and form shape module compactness and specialization, while task weighting alters functional allocation. Moreover, extracted functional tokens can be recombined through alternative pathways to reconstruct a learned composite computation without retraining, preserving their connectivity and learned weights. Our results suggest that sparsity-constrained optimization shapes not only parameter use but also the organization and reuse of learned computation.