Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explain that in single-cell perturbation data, individual cells often look too similar across different conditions, making it hard for models to correctly classify each cell. Traditional accuracy measures fail here because they reflect this overlap rather than true model quality. To fix this, the authors propose a method that averages predictions over all cells in a group to identify the correct perturbation at the population level. Their Classifier Discrimination Score (CDS) outperforms previous methods without needing model changes and works better especially when fewer cells are available. They suggest careful use of per-cell accuracy and similar scores for evaluating these models.
single-cell perturbation dataclassificationmacro-F1 scorepopulation profileClassifier Discrimination Score (CDS)pseudobulkPerturbation Discrimination Score (PDS)linear classifierMLPTransformer
Authors
Youssef Marrakchi, Davide D'Ascenzo, Sebastiano Cultrera di Montesano
Abstract
Most classification problems assume the classes are roughly separable, so that an individual sample can usually be assigned to one class. Single-cell perturbation data violates this assumption: two perturbations can produce different populations of cells while overlapping so much that an individual cell could belong to either. Per-cell accuracy then measures this overlap rather than model quality. We see this on Tahoe-100M and the Virtual Cell Challenge, where a linear classifier, an MLP, and a Transformer all plateau near macro-F1 0.2-0.3 even though almost every pair of perturbations is statistically distinguishable. The fix is to score perturbations across the whole population rather than cell by cell. We average a classifier's per-cell probability vectors over all cells of a perturbation to form a population profile, then rank candidate perturbations by this profile; we call the resulting score the Classifier Discrimination Score (CDS). Taking the top-ranked class recovers the winning perturbation. It needs no retraining, costs linear time in the number of cells, and recovers near-perfect identification from the same weak models. CDS differs from the pseudobulk-based Perturbation Discrimination Score (PDS) used in recent benchmarks only in where the average is taken, raw gene expression for PDS versus a learned discriminative space for CDS, and identifies the true perturbation more reliably on both datasets, with the gap widening as cells grow scarce. Because a metric that misranks the ground truth will misrank the models scored against it, per-cell accuracy and raw-pseudobulk scores should be used with caution when comparing perturbation models.