Few-shot learning emerges similarly across diverse data types
Convergent Emergence of In-Context Learning Across Modalities
Artificial Intelligence
Summary
Few-shot in-context learning means a model can learn a new task just by seeing a few examples in its input. The authors studied whether this ability appears in many types of data, like language, DNA, images, and music sequences. They found that this learning style does emerge in several different areas, and tasks that benefit from it in one type of data often do in others too. However, this pattern does not hold for every data type tested. This suggests a shared structure in how models learn from examples across different fields.
What this means in practice
- •For language model developers: Evaluate and improve few-shot learning capabilities across multiple data forms including text, genomes, and images using the identified task difficulty patterns.
- •For bioinformatics tool builders: Design genomic and protein sequence analysis tools that use few-shot in-context learning techniques shown to generalize from language models.
Authors
Nathan Breslow, Seungwook Han, Daniel Hyunsoo Lee, Aayush Mishra, Anqi Liu, Daniel Khashabi
Abstract
Few-shot in-context learning (ICL), the capacity of a model to infer abstract patterns from input-output examples provided in its prompt and apply them to new inputs, has been extensively studied in large language models trained for next-token prediction on human text. Recently, few-shot ICL has been demonstrated in autoregressive genomic models as well. This raises a question: does ICL emerge broadly across domains, and if so, what common structure is shared? To address both, we develop a controlled cross-modality framework that instantiates the same task suite in a variety of modalities to test what we call the Convergent Emergence Hypothesis: the idea that few-shot ICL, when it emerges, shares a common cross-modality difficulty profile - i.e., tasks that benefit from ICL in one modality tend to benefit in others. We show that paired-mapping ICL emerges across six modalities (language, genome, integer sequences, time series, images, and proteins), surpasses controlled baselines, and has correlated per-task effects across five of them. Together, these results provide support for the Convergent Emergence Hypothesis in some modalities, but not all.