LLM agents struggle to adapt when environment assumptions change
When Successful Strategies Fail: Adaptation to Environmental Novelty in Terminal Agents
Artificial Intelligence
Summary
Large language model (LLM) agents often solve tasks by assuming certain things about their environment. This study looks at what happens when these assumptions become wrong because the environment changes, but the goal stays the same. The authors created a tool called AGNI that tweaks environments in specific ways to see how well agents adapt. They found that while agents can notice something changed, they have difficulty figuring out what and adjusting their plans accordingly. Training agents with varied environments helps them adapt better and also improves their original performance.
What this means in practice
- •For robotics engineers: Improve robots’ ability to adapt when their surroundings or tool availability unexpectedly change during long tasks.
- •For virtual assistant developers: Develop assistants that detect and recover from wrong assumptions about user context or resources during complex interactions.
Authors
Janvijay Singh, Vaishnavi Shrivastava, Dilek Hakkani-Tur, Ece Kamar, Asli Celikyilmaz
Abstract
LLM agents increasingly solve long-horizon tasks by autonomously interacting with their environment. In doing so, their strategies rely on assumptions about that environment: which resources and tools exist, where they are located, and how they behave. When these assumptions no longer hold, reliable agents must detect the change and adapt while pursuing the same goal. We study this adaptation capability through environmental novelty: a change that keeps the task objective fixed while invalidating an assumption underlying an otherwise successful trajectory. We introduce AGNI, an automated pipeline that extracts trajectory-relevant assumptions, injects targeted environmental changes, and validates that the resulting novel tasks remain solvable. Across three terminal benchmarks, AGNI produces diverse novelties spanning resources, interfaces, constraints, and execution semantics. Evaluating multiple LLM agents reveals a substantial adaptation gap between base and novel tasks. Trajectory analysis suggests that agents often encounter evidence of the change but fail to diagnose its cause and revise their strategy. Finally, post-training for environmental novelty improves adaptation to held-out novel tasks while also improving performance on base tasks. Our results highlight a gap between task competence and adaptive capability and motivate environmental variation as a core dimension of agent training and evaluation.