Gene perturbation model rankings mislead due to evaluation pitfalls
A Pitfall in the Evaluation of Gene Perturbation Models
Machine Learning
Summary
Scientists use computer models to predict how genes respond when changed, but comparing these models hasn’t been consistent. This paper shows that common evaluation methods can give misleading results, especially when the same prediction is repeated multiple times, making some genes seem more affected than they are. The authors identify this problem and suggest a better way to compare models, finding a particular approach using gene features works best.
What this means in practice
- •For bioinformatics teams: Improve evaluation protocols of gene perturbation models to avoid false positives caused by repeated predictions in datasets.
- •For drug discovery teams: Use reliable gene perturbation modeling methods identified here to better predict genetic effects for target validation.
Authors
George Panagopoulos, Saman Omidi, Tsz Pan Tong, Jun Pang
Abstract
Gene perturbation models aim to predict transcriptional responses to unseen genetic interventions, but comparisons rely on heterogeneous evaluation metrics and differentially-expressed (DE) gene subsets. We investigate the consequences of these choices using eight methods across six single-gene perturbation datasets under a unified evaluation protocol. Varying the number of top DE genes reveals substantial instability in Precision-based rankings, with the winning method changing $86.7\%$ of the time, while rankings based on the rest of the metrics remain comparatively stable. Investigating this discrepancy exposes that repeating a single predicted expression profile across cell-level rows, a standard practice for many methods, can inflate significance estimates and produce excessively large predicted significant-DE sets. Specifically we found that $67.1\%$ of tested perturbation--gene pairs pass the significance threshold despite some of them having minuscule effects. We derive an inflation factor showing that replicated predictions can amplify the standardized Mann--Whitney statistic even after correction for tied ranks, without adding independent information. This artifact changes Precision mainly as the number of DE genes increases because for smaller cutoffs, most of the genes are already significant due to the effect-based ordering. Taking this pitfall into account in our benchmark comparison, we find that a random forest using Gene Ontology features achieves the best overall results.