Benchmark evaluates AI agents on real-world bioimage analysis
BIABench: Evaluating AI agents on real-world bioimage analysis tasks
Artificial IntelligenceComputation and Language
Summary
Analyzing biological images is hard for AI because the images are often large and complex, including 2D, 3D, and video data. The authors created BIABench, a test with 16 tasks taken from real biological studies, where AI agents must analyze data and produce reliable results. They found that AI can handle simple 2D tasks well, but struggles with complex 3D and time-based images. The benchmark also shows AI methods are often inconsistent, and it provides a way to measure and improve AI performance on these problems.
What this means in practice
- •For bioimage analysis engineers: Test and compare AI algorithms for end-to-end biological image analysis with real-world data and standardized scores.
- •For medical image software developers: Use the BIABench benchmark to improve software reliability in processing diverse biological imaging modalities including 3D and time-lapse data.
Authors
Zixuan Pan, Davide Panzeri, Lukas Johanns, Marilin Moor, Yu Zhou, Hedi Peterson, Yiyu Shi, Jianxu Chen
Abstract
Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end. Such analyses are hard for agents because 2D images, 3D volumes and time-lapse sequences are often too large to read as context, so an agent must choose and run an analysis through code, specialized software and rendered views. Published studies make this capability testable, because each pairs raw images with a peer-reviewed result. We introduce BIABench, a benchmark of 16 tasks reconstructed from published biological studies that retain their scientific questions, imaging data and ground truth. The tasks span eleven analysis subtasks and modalities from H&E histology to single-molecule localization microscopy. Each submission receives an outcome score, which compares the output files with the ground truth using field-standard metrics, and a process score, in which a vision-language model judges method choice and quality control against an expert-written rubric. We evaluated general-purpose and biology-specific agents across several language models, with repeated runs of every task. Routine two-dimensional tasks were solved well, but on some tasks that added a third dimension or a time axis no agent scored above 0.19. Neither biological specialization, stronger models nor detailed expert instructions closed this gap. The agents were also unreliable, with scores varying more between repeated runs of one agent than between different agents, and without ground truth a correct run could not be told from a wrong one by its process score or by the time spent. Released openly with its data and code, BIABench provides a verifiable framework for evaluating, and eventually training, agents for reliable long-horizon bioimage analysis.