Frontier agents improve scientific models mainly by engineering tricks
Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Improving scientific models with AI often needs deep knowledge from the field, not just machine learning skills. The authors created a test called Agent4Sci to see how well current AI agents can enhance top scientific models across many areas. They found that today's best agents do make improvements, but mostly by using clever engineering methods rather than true scientific insight. Adding some tools that help with research brainstorming and experiment planning boosted these agents’ performance and creativity. Agent4Sci helps track progress in moving AI agents from simple engineering toward deeper scientific understanding.
What this means in practice
- •For ml engineers: Improve scientific ML models by integrating lightweight tools for research ideation and experiment scheduling to boost model quality.
- •For science software developers: Develop smarter ML-for-science systems that monitor and assist with domain-specific experiments leveraging agent4sci benchmarks.
Authors
Ming Zhong, Stacy Li, Nicholas Carlini, Matthew Jagielski
Abstract
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).