How Much Do RF Drone Benchmarks Overstate? A Controlled Study and Theory of Data Leakage in UAV Signal Identification
2026-07-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explain a problem with how some drone detection studies test their models using radio signals. They show that when parts of the same recordings are used both for training and testing, the model seems smarter than it really is because it just memorizes rather than truly learns. Their experiments confirm that this causes artificially high accuracy numbers, which drop to random guessing levels when proper testing methods are used. This means previous high accuracy claims might be misleading due to data leakage.
Radio-frequency (RF) sensingCounter-unmanned-aerial-system (counter-UAS)Cross-validationData leakageCover's function-counting theoremBayes accuracyDroneRF datasetRecording-grouped evaluationNuisance variationMacro-F1 score
Authors
David Shulman
Abstract
Radio-frequency (RF) sensing is a central modality for counter-unmanned-aerial-system (counter-UAS) defence because it exploits the control, telemetry, and video links between a drone and its operator. Reported accuracies for RF-based drone detection and identification are often very high, but many are obtained using cross-validation that splits a small number of continuous recordings into short segments. This can place near-duplicate slices of the same recording in both training and test partitions, creating data leakage. We study this leakage problem through theory and measurement. We formalise the optimism of segment-level cross-validation and show, using Cover's function-counting theorem, that a classifier can exactly memorise the recording-to-label map when the number of independent recordings, R, is small relative to the feature dimension, d. In particular, this can occur when 2R is less than or approximately equal to d. Under these conditions, naive accuracy approaches 1, and the inflation gap approaches 1 - ACC*, where ACC* is the Bayes accuracy. The inflation eases only once R grows beyond this separability threshold. A controlled synthetic experiment with 10 seeds confirms the predicted curves: naive balanced accuracy rises from the Bayes level toward 1.0 as recording-specific nuisance variation grows, while honest recording-grouped evaluation declines to chance, with a gap reaching about 0.5. On the public DroneRF dataset, pooled leave-one-recording-out cross-validation shows drone type identification, AR versus Bebop, collapsing from a naive macro-F1 of 0.74 to 0.46, the two-class chance level. A leakage-pathway ablation attributes essentially all of the inflation to segment-level leakage.