SkillShift benchmark measures web agent skills under changing conditions
SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents
Artificial IntelligenceComputation and Language
Summary
Web-based AI agents use skills to complete complex tasks by learning from past experience. The authors point out that these agents often fail when the online environment changes, like when web pages look different or functionalities update. They created SkillShift-Bench, a set of tests that simulate these changes to see how well agents keep performing. Their experiments show agents struggle more under certain types of changes, especially when execution conditions shift. The authors also propose a way to adapt agents during use without retraining all their skills.
What this means in practice
- •For web automation developers: Test and improve the robustness of automation tools under evolving website designs using SkillShift-Bench's controlled environment shifts.
- •For customer support bot teams: Enhance chatbot skill adaptability for handling updated web service interfaces without full retraining, reducing downtime during changes.
Authors
Bo-Yu Chen, Zhi Zhou, Yu-Feng Li
Abstract
Skill learning enables web agents to reuse prior experience for long-horizon interactive tasks. However, learned skills are typically evaluated in closed environments, overlooking the fact that real-world web deployment involves continuous shifts in invocation context, perceptual grounding, and execution feasibility. To this end, we introduce SkillShift-Bench, the first benchmark for evaluating skill learning with controlled open-environment shifts. SkillShift-Bench simulates Contextual Shift, Perceptual Shift, and Execution Shift across three self-hosted web systems, and evaluates agents with diagnostics covering task success, interaction cost, skill use, and skill failure. Comprehensive experiments reveal systematic robustness degradation under controlled shifts, characterized by lower success rates, higher interaction costs, and more frequent skill failures, with Execution Shift exerting the strongest pressure. We further formulate shifted skill execution as a nonstationary local executability problem and provide both a dynamic-regret analysis and a practical test-time adaptation method, demonstrating the feasibility of improving learned-skill execution without relearning the skill library. Code, tasks, environments, documentation, and the leaderboard are available at https://skillshift-bench.github.io/.