Cognitive decoupling improves learning across different tasks
Cognitively Decoupled Meta-Induction for Out-of-Distribution Generalization
Machine LearningArtificial Intelligence
Summary
Machine learning models usually struggle when they face questions very different from what they’ve seen before. The authors created a new method called CoDI to help models learn general rules across many tasks instead of just matching examples closely. This approach helps the model understand and apply broader patterns, making it better at handling new, unfamiliar problems. Their experiments showed that CoDI performs better on several tests both when data is similar and when it is different from training examples.
What this means in practice
- •For machine learning engineers: Improve model robustness for tasks where training and test data differ significantly by inducing generalizable rules.
- •For natural language processing developers: Enhance language models' ability to generalize beyond seen examples for diverse downstream tasks.
Authors
Yuqian Wang, Yuanzheng Wang, Chaojun Xiao, Ke Xu, Bingxiang He, Jinghan Tan, Zijun Chen, Lu Chen, Yi-Xin Huo, Maosong Sun
Abstract
In-context learning (ICL) tackles downstream tasks conditioned on demonstrations. This paradigm relies heavily on neighbor-based matching between queries and demonstrations, limiting out-of-distribution (OOD) performance. For better generalization, rule-based inductive reasoning derives task-level regularities, which we term task-level inductive representations (TIR), before directly answering. However, existing methods lack generalizable inductive ability across diverse tasks. Inspired by meta-learning, we propose Cognitive-Decoupling meta-Induction (CoDI) for TIR-based inductive reasoning on general tasks. CoDI comprises a Cognitive-Decoupling Framework for query-blind TIR induction, and a Dual-Loop Meta-Inductive Learning method for cross-task inducer optimization. Empirically, CoDI achieves substantial performance gains on five ID and OOD benchmarks, and preserves OOD gains without explicit induction at inference. Code is available at https://github.com/Yuqian-Wang2277/codi-meta-induction.