StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models
2026-08-24 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study how big language models learn new tasks when given only a few examples, known as few-shot in-context learning (ICL). They note that current methods just copy examples and don’t clearly learn the underlying task rules, making the models sensitive to which examples are chosen. To better evaluate this, the authors created StrategyBench, a benchmark that tests how well models can summarize task rules (strategies) and use them on new problems. Their experiments show that performance varies depending on the type of task and how the strategy is generated and applied.
few-shot learningin-context learninglarge language modelstask adaptationtask strategiesbenchmarkingBIG-Benchstrategy inductionmodel evaluation
Authors
Jinghan Tan, Yuanzheng Wang, Lu Chen, Zijun Chen, Yuqian Wang, Maosong Sun
Abstract
As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.