Coding agents struggle to adapt when tasks and environments change
SWE-Shift: Evaluating Coding Agents in Evolving Workspaces
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Software coding often changes halfway through a task, but many coding AI tools assume everything stays the same. The authors created SWE-Shift, a test to check how well AI coding helpers manage when the task or environment updates while they work. They found all tested systems perform worse with these mid-task changes, showing that even if an AI finishes the original job, it often fails to keep earlier work or adjust correctly. This means current AI coding tools still have trouble handling real-world changing coding situations.
What this means in practice
- •For software development teams: Evaluate and improve AI coding tools’ ability to handle changing tasks and environments during active development.
- •For software testing engineers: Use SWE-Shift scenarios to test how well coding assistants maintain code correctness when dependencies or requirements change mid-task.
Authors
Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo, Renrui Zhang, Hongsheng Li
Abstract
Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.