Meta learned networks link rapid learning to parameter addition
Linking Rapid Learning to Compositional Structure in Neural Networks: Evidence from Additive Analogies in Meta-Learned Parameter Spaces
Machine LearningComputation and Language
Summary
Neural networks usually need lots of practice to learn new tasks, but humans can learn quickly from just a little experience. This paper looks at special neural networks trained to learn faster by seeing many similar tasks first. The authors found that the networks' internal settings can be combined like puzzle pieces to create new abilities without extra training. This means the networks organize learning in a way that reflects the symbolic relationships between tasks, helping them pick up new things quickly.
What this means in practice
- •For machine learning engineers: Design meta-learning models that combine trained parameters arithmetically to quickly specialize on new tasks without retraining.
- •For ai system architects: Create adaptable AI systems whose internal configurations reflect task relationships through vector addition, enabling efficient function reuse.
Authors
Abigail L. Tenenbaum, R. Thomas McCoy
Abstract
Humans can learn rapidly from limited experience, while neural networks typically require extensive training. In this work, we investigate what properties of a neural network can enable it to learn rapidly. We study neural networks that have been optimized via meta-learning, a technique that exposes a network to many related functions to facilitate its subsequent learning of similar functions. Prior work has shown that these meta-learned networks can learn new functions rapidly, but how they do so remains unclear. To investigate their internal structure, we take inspiration from classic work demonstrating additive relationships among the learned vectors that neural networks use to represent words (e.g., "king" minus "man" plus "woman" approximates "queen"). Analogously, we look for additive relationships among a meta-learned network's "representations" of different functions; when a meta-learned network rapidly learns to implement a specific function, all of its adapted parameter values can be concatenated into a single vector that we treat as representing the function. We find that such parameter vectors can be added and subtracted according to symbolic relationships among the functions they represent, yielding new parameter vectors that successfully represent the functions predicted by the corresponding symbolic manipulations: the networks specified by these artificially recombined vectors actually implement the intended functions without any direct training on them. Our results suggest that meta-learning can tune a network's starting point to organize subsequent learning along systematic, functionally meaningful trajectories through the space of possible parameter vectors, providing a geometric basis for realizing inductive biases in neural networks.