Benchmark shows agents struggle with real business task delivery
FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
Artificial IntelligenceMultiagent Systems
Summary
Many business requests don’t clearly explain what is needed because workers rely on experience they can’t fully describe. The authors created a benchmark called FDE-Bench using real cases where engineers have to talk to customers to figure out missing details and improve solutions step-by-step. They tested different AI agents on these tasks, but none performed as well as human engineers. This means current AI still finds it hard to fully handle complex business tasks where requirements are unclear.
What this means in practice
- •For business solution developers: Evaluate and improve AI agents’ ability to clarify requirements and deliver solutions for complex, real-world business tasks.
- •For customer support automation teams: Develop better dialogue systems that can handle unclear customer requests by iteratively refining requirements through interaction.
Authors
Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, Qianlong Wang, Zengle Ge, Xiyu Yan, Yuheng Cheng, Jinhui Ren, Guokai Chen, Jiaqun Liu, Haoran Li, Qianhui Liu, Annan Li, Jianmin Wu, Dawei Yin, Dou Shen
Abstract
Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at https://baidubce.github.io/FM-Agent/FDE-Bench/.