Dataset precision depends on true event rarity not just detector accuracy
Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets
Machine LearningArtificial Intelligence
Summary
When machine learning datasets are created by detectors that pick items from large pools, the accuracy depends heavily on how common the real items are in the pools, not just on how good the detector is. The authors show with real data that many so-called detections are actually false (phantoms), especially in some pools. This means datasets have a silent contamination that behaves like a second signal, not simple noise. They also show how some common statistics can be misleading if this contamination is not accounted for.
What this means in practice
- •For machine learning engineers: Improve dataset construction by accounting for true event prevalence to reduce silent false positive contamination.
- •For data annotation teams: Identify pools with high phantom rates to better focus manual verification efforts and improve dataset label quality.
Authors
Jia Huang, Yankai Wan, Yangjun Ou
Abstract
Many ML datasets are constructed by running a detector, heuristic, or model over candidate pools; accepted items become labels. Dataset precision is then governed by true-positive prevalence in each pool via Bayes, not solely by detector quality. Using one instrument and period, we hold a detector-defined event dataset plus an independent official index labeling every detected item as real or phantom. One detector, three pools yield phantom rates 81.7%, 9.0%, and 0.0%. Transferring precision from the two high-rate pools to the low-rate pool predicts 0.955 versus measured 0.183, a +422% error; the Bayes expression predicts all three within 3.3%. The detected response curve is an exact convex combination of a true-event and a phantom component (residual 1.1e-16), with phantoms outnumbering true events 473 to 308, so contamination is a second signal with detector-inherited shape, not additive noise. Contamination direction depends on the estimator: on identical windows one statistic is diluted and another inflated because its denominator is also contaminated. A common normalization turns the estimator into a mean of ratios whose expectation need not exist; on the same 335 events it returns 0.40 where the well-defined estimator returns 0.10.