Sciencecua builds evolving executable skills from scientific software docs
ScienceCUA: Resource-to-Skill Construction and Execution-Guided Skill Evolution for Scientific Computer Use Agents
Artificial Intelligence
Summary
Using scientific software can be tricky because manuals and websites don't always match the software version, and they don't cover every task people need. The authors introduce ScienceCUA, a system that reads software documentation to create reusable skill sets for performing tasks. Then, it learns from which attempts succeed or fail in real use, improving these skills over time. They also made new benchmarks to fairly test such systems. This helps general-purpose agents use complex scientific software more reliably.
What this means in practice
- •For scientific software developers: Automatically generate and update task-specific skills from documentation to assist users navigating complex scientific tools.
- •For automation engineers: Build adaptive agents that improve use of scientific software by learning from execution success and failure to handle evolving software.
Authors
Daiqiang Li, Jingyi Peng, Rui Hu, Jia Li, Xuyang Liu, Guanren Qiao, Yujia Xiang, Qiuzhuang Sun, Zhiping Xu, Mingrui Luo
Abstract
Computer-use agents for scientific software require application-specific expertise. Much of this expertise is documented in human-authored software manuals and web pages. However, turning these resources into executable, reusable skills remains challenging due to mismatches between documented and deployed software versions and differences in scope between documentation coverage and task requirements. We introduce ScienceCUA, a two-stage framework combining resource-to-skill construction with execution-guided skill evolution. First, ScienceCUA identifies tasks supported by the software from its documentation, infers operational trajectories, and generalizes them into reusable Skills. Command catalogs, syntax constraints, and shortcuts are organized separately as References for on-demand lookup. Then, ScienceCUA uses execution feedback to drive skill self-evolution by contrasting successful and failed trajectories for the same task, identifying deficiencies, revising existing skills or adding new ones, and updating the skill library only after repeated validation in real environments. To enable reliable evaluation without benchmark noise or task leakage, we introduce two improved benchmarks. ScienceBoard-Verified provides an audited and corrected basis for measuring task performance, while ScienceBoard-VE separates skill evolution from final testing through disjoint evolution and test tasks and further supports native-agent evaluation. Experiments show that the initial Skills and References improve average task success rate by up to 7.23 percentage points over the base model, while skill self-evolution increases the total gain over the baseline to as much as 17.12 percentage points. ScienceCUA turns software documentation into an evolving set of executable skills and queryable references, offering a practical way for general-purpose agents to use scientific software reliably and at scale.