Benchmark measures autonomous AI model builders in biomedical research
BioXArena-Pro: a research-grade benchmark for autonomous biomedical machine learning agents
Machine Learning
Summary
Building computer models to understand biology is tricky, and existing tests for AI tools either have too few challenges or leak answers. The authors created BioXArena-Pro, a set of 64 tough biomedical tasks from real studies, to better test AI model-building agents. They found that which agent framework is used matters a lot, and agents that adapt their methods to the data do better than just copying old recipes. Pretrained models help some tasks, showing the value of prior biological knowledge in AI.
What this means in practice
- •For biomedical ai developers: Test and improve autonomous AI agents across a wide range of biomedical modeling challenges with data resembling real-world biological tasks.
- •For healthcare data teams: Assess AI tools for building predictive models on complex biomedical data from different species or patient groups to support research decisions.
Authors
Loka Li, Duzhen Zhang, Xingbo Du, Leonard Song, Kun Zhang, Le Song
Abstract
Coding agents are increasingly proposed as autonomous builders of biomedical models, but existing benchmarks are small, near saturation or open to leaked information. We present BioXArena-Pro, 64 research-level modelling tasks from eight biomedical domains, each rebuilt from a published study; most hold out whole biological units such as species or cohorts. Across 22 agent configurations, the best scores 55.1 of 100, and the scaffold matters almost as much as the model. We re-ran a published method for every task: where the published recipe applies unchanged, the best agent and the methods tie; on rebuilt tasks the methods trail by 8.2 points, suggesting that agents gain by adapting to new data. Pretrained models (including biological foundation models) matter where the strongest agent uses them: forbidding them decreases its score on those tasks by 9 points. Every task was audited in several AI-assisted rounds and a final human review.