Language models can learn to ignore bad advice and trust good help
Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?
Computation and Language
Summary
Sometimes language models get confused when given bad advice, even if some advice is useful. The authors study how well models can tell when to follow helpful instructions and when to ignore misleading ones. They test this by giving the same task but changing how reliable the advice is, and train models to be better at deciding when to listen. Their work shows models can learn to use good guidance while avoiding bad, which helps with other types of outside information, too.
What this means in practice
- •For software engineers: Improve AI assistants to selectively trust external plugins or APIs by learning to ignore unreliable inputs automatically.
- •For data platform operators: Enhance data pipeline systems that use language models by making them robust to corrupted or misleading external data sources.
Authors
Minghan Wang, Boyuan Wang, Jinhang Zuo, Yuxin Tao, Fang kong
Abstract
Agent harnesses often improve language models with human-designed workflows, but as models grow more capable, unreliable guidance can increasingly constrain their execution. We call the ability to benefit from useful guidance while overriding unreliable guidance thinking outside the box. We introduce Box$^2$-Bench, which holds the model and task fixed while varying workflow reliability to isolate how models regulate their reliance on guidance. On Box$^2$-Bench, frontier models often benefit from reliable guidance but remain vulnerable when it is misleading or becomes unreliable. To test whether this capability can be learned, we train two open-weight models using bad workflows, reserving good workflows for evaluation. We explore two complementary training strategies: counterfactual supervised fine-tuning improves robustness, while outcome-based reinforcement learning can shift the balance toward greater use of helpful workflows. We further find that this behavior extends beyond workflows to other forms of external information, improving peer correction and robustness to corrupted memory. Together, our results identify selective reliance on fallible external information as a dimension of agent reliability not captured by task performance alone.