AI robustness tested on biomedical images from varying lab conditions
BatchBench: Benchmarking AI Robustness to Batch Effects and Domain Shift in Biomedical Images - A JUMP Cell Painting Leaderboard
Machine LearningArtificial Intelligence
Summary
AI systems often struggle when the images they analyze come from different labs or experiments because the pictures can look quite different. The paper presents BatchBench, a kind of test leaderboard that checks how well AI can handle these differences in biomedical images. It uses a large dataset called JUMP-CP, which has many examples showing changes caused by different testing batches. This makes it a good way to see if AI can still perform well even when conditions change in the lab.
What this means in practice
- •For hospital data teams: Evaluate AI diagnostic tools to ensure consistent performance across images from different labs or machines.
- •For pharmaceutical research teams: Compare AI models for analyzing drug effects in cell images from varied experimental batches to improve reliability of findings.
Authors
Sonja Topf, Ana Sanchez-Fernandez, Günter Klambauer
Abstract
Despite recent advances, current AI systems remain highly susceptible to domain shifts, particularly in biomedical imaging, where images are usually acquired under substantially different experimental conditions. Although recent domain adaptation methods have made substantial progress, significant performance gaps under domain shift persist. To measure progress in this regard, we introduce BatchBench, a leaderboard that evaluates and compares the robustness of AI systems on biomedical images from unseen experimental batches and data sources. BatchBench is built on JUMP-CP, one of the largest publicly available biomedical imaging datasets. Generated using the Cell Painting assay, JUMP-CP exhibits pronounced batch effects and domain shifts, making it a challenging testbed for evaluating robustness to distributional changes.