Papers for

software developers building continual learning systems

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Neural networks evolve architecture while learning to improve efficiency

FlowGenesis: Representation-Driven Self-Organization of Neural Architectures

Abstract: Deep neural networks typically rely on architectures specified before training, despite the continuous evolution of their internal representations. We introduce FlowGenesis, a framework in which neural architectures adapt through neuron birth and pruning guided by Representation Flux, a local statistic capturing changes in minibatch mean activations. Neurons exhibiting persistently high Flux undergo splitting designed to approximately preserve the network function, while neurons with low Flux and low activity are pruned. This mechanism allows network capacity to evolve without prescribing a target architecture or performing a separate architecture search. Experiments on image and text classi- fication show that networks initialized with different sizes approach similar final neuron-count ranges within each dataset while maintaining comparable predictive performance. We further extend FlowGenesis to continual learning, where representation-guided structural adaptation reduces the neurons and parameters allocated to successive tasks while maintaining accuracy comparable to fixed-width Progressive Neural Networks in a task-incremental learning setting. These findings demonstrate that neural architectures can adapt through feedback between representation learning and structural change, suggesting an alternative to prescribing network capacity before training.

Mon 26 OctMachine Learning
The gist
Neural networks often start with a fixed setup before training, even though their inner workings change during learning. The authors present FlowGenesis, a method that lets the network grow new neurons or remove unnecessary ones based on how much each neuron’s activity changes. This way, the network’s structure adapts on its own without needing designers to pick the size in advance. Tests show this adaptive network keeps good performance across image and text tasks while ending up with similar sizes regardless of the starting point. It also helps networks learn new tasks with fewer neurons while preserving accuracy.
Open → 2610.35108v1