EvoProbe improves testing of large language model agents with evolving tasks
EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents
Artificial Intelligence
Summary
Static tests for large language models often become too easy or outdated, making it hard to understand where these models struggle. The authors propose EvoProbe, a method that continually creates new and harder tasks based on detailed feedback from earlier tests. By analyzing where the models fail or succeed in smaller steps, EvoProbe updates the tasks to better probe their abilities. Experiments in cybersecurity tasks show that this approach reveals weaknesses not visible with simple overall scores.
What this means in practice
- •For ai system developers: Test and improve AI agents by generating evolving tasks that identify specific weaknesses beyond aggregate scores.
- •For security software teams: Use iterative task updates to evaluate and strengthen AI models involved in cybersecurity problem solving.
Authors
Qianhong Guo, Wei Xie, Xiaofang Cai, Shuoyoucheng Ma, Chengxin Hu, Baosheng Wang
Abstract
Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8\% in mean Outcome and 40.0\% in mean Pass\textasciicircum{}3. These results support using fine-grained diagnostics to guide iterative task updates and probe agents' current capabilities and limitations.