AgentProofBench tests web agents for reliable rule following and execution
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
Multiagent SystemsArtificial IntelligenceSoftware Engineering
Summary
Web agents need to follow rules that can change and still perform tasks correctly. The authors created AgentProofBench, a tool that checks if web agents keep working well even when rules or starting points change. They tested many tasks across different web services, showing that their benchmark can spot when agents fail or succeed in following rules. This helps measure how well verification methods protect against mistakes during web agent execution.
What this means in practice
- •For web developers: Evaluate and improve web agent reliability by testing rule adherence and execution stability under changing conditions.
- •For automation engineers: Use AgentProofBench to verify reusable automated workflows across multiple web services to catch regressions early in development.
Authors
Jingjie Ning, Xueqi Li, Yi Lu, Yibo Kong
Abstract
Reusable Web-agent verification requires a rule to follow changing requirements and support subsequent execution. We introduce AgentProofBench, which connects paired tests of a frozen acceptance rule to execution from identical saved openings. The resource spans 97 tasks, 79 families, and four Web services. Experiments with DeepSeek Flash/Pro establish two linked measurements. Across 48 families, complete-panel and decision-replay tests retain 82/96 Flash and 79/96 Pro source programs; paired reuse-rebuild BA intervals are [-5.47, 1.57] and [-10.16, 0.00] percentage points. Across 22 execution families, deployment-state panel correctness separates program assignments whose two saved openings are both delivered correctly: 32/33 assignments with error-free panels, versus 1/11 with panel errors. All reference-defined program-gate regressions and withheld deliveries occur in the latter group. Neutral continuation attains the highest correct delivery under the tested generic feedback, providing a concrete baseline for measuring the value added by a gate. Released instructions, states, programs, reference rules, and trajectories connect each test failure to its execution consequences. AgentProofBench makes reusable verification assessable at both the requirement and policy levels.