CellChain benchmark assesses AI reasoning across biological data scales
CellChain: Benchmarking Cross-Scale Evidence Use for Perturbation Reasoning
Databases
Summary
Predicting how cells respond to changes is complicated because it involves different levels of biology, from molecules to how cells behave. The authors created CellChain, a test set with many examples that check how well large AI models use different kinds of biological evidence to make predictions. They found that giving AI more evidence doesn’t always improve its answers and that AI models can change answers in ways not obvious from overall accuracy scores. CellChain helps compare AI models by showing how their predictions depend on the specific biological evidence they use.
What this means in practice
- •For biotech development teams: Evaluate AI models for drug response predictions using multiple biological evidence types under controlled conditions.
- •For bioinformatics platform developers: Integrate CellChain benchmark to test improvements in model reasoning across molecular to phenotypic biological data.
Authors
Xue Xia, Yue Wang, Shuizhou Chen, Lang Yu, Hao Wu, Wenxuan Huang, Chengkai Yao, Zhangyang Gao, Siqi Sun
Abstract
Recent advances in foundation models and large-scale cellular perturbation datasets have driven growing interest in using LLMs to predict and reason about cellular responses. Biological reasoning in this setting involves relating molecular mechanisms, cellular responses, and functional outcomes across scales. Evaluating foundation models therefore requires assessing not only whether their predictions are correct, but also how those predictions depend on the biological evidence provided. We introduce CellChain, an experimentally grounded benchmark comprising 590 biological cases and 2,360 matched evidence-condition instances across gene-dependency prediction, cytokine-axis identification, and drug-sensitivity ranking. CellChain links perturbation experiments and external biological resources through a provenance-aware evidence graph spanning molecular, pathway, transcriptional, and phenotypic information. Task-specific conditions vary evidence coverage, composition, or assignment while preserving questions, candidate answers, reference targets, and scoring rules. Across 18 foundation models, the highest accuracies under prespecified target-evidence conditions are 57.5%, 75.5%, and 70.8%, respectively, and broader evidence coverage does not consistently improve accuracy. On the drug-sensitivity benchmark cohort, matched reference histories outperform swapped histories by an average of 13.0 percentage points in cluster-weighted accuracy. Decision-transition analysis separates corrections, newly introduced errors, and wrong-answer switches, revealing substantial answer changes hidden by small net accuracy differences. CellChain provides controlled comparisons for attributing changes in model predictions to prespecified expansions of biological context, including changes in evidence coverage, composition, and biological assignment.