AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
2026-07-06 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors explain that current tests for language agents (like advanced AI models) are too simple and don’t reflect real-world challenges, where information can be messy and tools need to be found and combined on the fly. They created AgentGym2, a new way to test these agents with tasks that mimic real-world situations more closely. Their tests showed that even the best models today, like Gemini and GPT-5, have trouble handling these realistic problems. This means current AI agents are not yet fully ready for complex, real-world use. The authors highlight the need for better evaluation methods to improve AI agents.
Language agentsLLM agentsBenchmarkingTool discoveryRobustnessNoisy dataEnd-to-end tasksExplorationGPT-5AI evaluation
Authors
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
Abstract
Language agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications.