New benchmark reveals gene-specific limits in predicting cell responses

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Artificial IntelligenceMachine Learning

Summary

Predicting how cells react when specific genes are changed is important but complicated because models sometimes just learn common reactions rather than gene-specific effects. The authors created AmbiModBench, a new way to test these predictions that separates shared patterns from unique gene effects and checks how well predictions cover different gene areas. Their tests showed that many existing methods score well by capturing general trends but fail to identify individual gene responses clearly. This benchmark helps find methods that truly understand gene-level effects, improving how experiments in single-cell genomics can be prioritized.

What this means in practice

Authors

Sikai Huang, Zhiwen Yang, Kai Yu, Jiayuan Chen, Stan Z. Li

Abstract

Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.