Detection methods highlight mislabeled Google Play apps but do not improve predictions
Identifying Suspected Mislabeled Apps in Google Play Application Removal Prediction: An Empirical Comparison of Label Noise Detection Methods
Cryptography and Security
Summary
The problem is that apps removed from Google Play can disappear for different reasons, but the data only says if they were removed or not, making the labels noisy. The authors tested three ways to find apps that were likely mislabeled and found a small group where the label didn’t match the app’s features well. Removing these apps didn’t help improve prediction models and confirmed mislabeled apps appear less among those verified by security services. The value of these methods is in understanding the noise, not in improving removal predictions.
What this means in practice
- •For mobile app developers: Identify suspicious labeling patterns in app status data to better understand challenges in automated app removal prediction models.
- •For cybersecurity analysts: Use label noise detection insights to better interpret discrepancies between app behavior and store removal labels in security assessments.
Tested on one dataset.
Authors
Deborah Dobles Montalvan, F. Mohsen, H. de Weerd
Abstract
Models that predict which Google Play apps will be removed are trained on labels that record only whether an app was still in the store at a later observation. A disappeared app is labeled removed and a present one stable, but neither records why. A voluntary withdrawal and a policy takedown both produce removed, and an uncaught spam app keeps stable. This work calls that mismatch label noise. Three detectors from different methodological families are applied to the 870,514 apps of Mohsen, Karastoyanova, and Azzopardi (2022): Isolation Forest, flagging apps unusual in the feature space, Neighborhood Disagreement, flagging apps whose nearest neighbors carry the opposite label, and Prediction Inconsistency, flagging apps a classifier labels differently from the data. The apps flagged by all three, the overlap, number 7,598 at default settings and are the strongest mislabeling candidates. Two questions follow. First, does removing flagged apps improve the model? It does not. No detector, overlap, or union beats the baseline, and the loss grows with the number removed. Second, do flagged apps appear less often than expected among apps whose label VirusTotal and Quark Engine confirm? Among confirmed removals they do, falling to 0.43 times the expected rate as the threshold tightens, while an excess on the stable side disappears once the age of the scanned apps is accounted for. A model trained on only the 3,021 trainable overlap apps reaches a test AUC of 0.2518, far below chance, so the relationship between features and labels there runs opposite to the rest of the data. The flagged apps run wrong in both directions: abandoned apps that resemble spam carry stable, while apps that look healthy carry removed. The value of the detectors lies in characterizing this label noise. They locate a small set of candidates they cannot profitably remove.