Agents build better financial benchmarks for varied scenarios
FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
Artificial Intelligence
Summary
Financial tasks that use computers can be very different and complex, but creating tests to check how well these tasks are done usually requires a lot of manual effort. The paper’s authors created a big set of test requests and a way to check the quality of these tests. They then built a system made up of several cooperating agents to automatically create these financial computer tasks. Their system was much better than previous automatic methods, creating more and higher-quality test tasks. This shows that intelligent agents can help build better tests to evaluate financial computing tools on many different situations.
What this means in practice
- •For financial software developers: Generate diverse and realistic financial tasks automatically to test and improve financial computing tools against varied real-world conditions.
- •For enterprise software testers: Use agent-generated financial tasks to better evaluate workflow software under different runtime setups without manually coding test cases.
Authors
Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie
Abstract
Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agent, evaluation tasks remain largely manually constructed, limiting scalable coverage of real-world financial scenarios. Then, can agents autonomously construct diverse CUA evaluation tasks for financial scenarios? Evaluating this capability poses three key challenges: scenario coverage of construction requests, fair comparison across construction methods, and reliable assessment of generated task quality. To solve these, we introduce FinCUABuildBench, a benchmark for evaluating financial CUA task construction, featuring: (i) 576 construction requests covering 24 financial workflows and three types of runtime variation; (ii) standardized input, budget, and output specifications; and (iii) a task qualification mechanism based on execution tests and quality checks. We further introduce FinCUABuildAgent, a multi-agent system for automatically constructing dynamic financial CUA evaluation tasks. It consists of three modules that jointly construct tasks, environments, and validators. On FinCUABuildBench, under the same model backbone, existing agent-based construction methods achieve strict qualification rates of only 1.3-8.3%, while FinCUABuildAgent reaches 31.3%. Downstream evaluations further show that the constructed tasks can effectively differentiate CUA task-execution capabilities. These results demonstrate that agents can autonomously construct financial CUA tasks with meaningful evaluation value, offering a practical path toward broader evaluation coverage in financial scenarios. Code: https://github.com/FengxianJi/FinCUABuild