Language models identify key features for concept learning quickly
An Algorithmic Account of Few-Shot Concept Induction in Language Models
Computation and Language
Summary
Figuring out what features matter when learning something new from just a few examples is hard for both humans and AI. The authors propose a method called Amortized Direction Accentuation (ADA) that helps language models focus on the important parts of examples to understand the concept behind them. This method uses prior knowledge about which features often matter to create a summary that highlights what is relevant. They tested ADA on tasks where models had to learn concepts from few examples and found it helps explain how models and minds might do this. This could improve how AI understands contexts and learns quickly from limited data.
What this means in practice
- •For natural language processing teams: Improve language model performance on tasks requiring understanding new concepts from limited examples by using ADA to focus on relevant features.
- •For cognitive computing developers: Design AI systems that better mimic human-like concept learning by integrating ADA to identify contextually relevant features efficiently.
Authors
Michael A. Lepori, Laila Johnston, Aalok Sathe, Zhuonan Yang, Roman Feiman, Ellie Pavlick
Abstract
Both models and minds not only encode features that describe an entity, but can specifically represent which features of that entity are relevant for a given context. This core capability underlies in-context learning in language models and a variety of cognitive phenomena in minds, yet it is unclear how intelligent systems identify and promote relevant features. In this work, we propose Amortized Direction Accentuation (ADA), an algorithmic hypothesis that provides an explanation for how relevance is determined in the context of few-shot concept induction --- the ability to infer a latent concept from a small number of positive examples. Under ADA, concept induction proceeds by forming a summary representation of the examples in a feature space that is shaped by amortized knowledge about which features tend to be useful for finding commonalities between items. We investigate and validate ADA in language models using a few-shot concept induction task. This work provides a novel perspective on the mechanisms supporting contextualization in language models and a promising hypothesis for a core aspect of cognition.