Dual perspective test input prioritization improves dnn fault detection
When Ambiguity Meets Atypicality: Dual-Perspective Test Input Prioritization for DNNs
Software Engineering
Summary
Deep neural networks can sometimes make unexpected errors, which is a problem for software relying on them. The authors propose a new way to find test cases that are likely to cause errors by combining two ideas: how unsure the network is about its decision and how unusual the input is compared to what it normally sees. Their method, called DuFP, uses these ideas together to better spot tricky inputs that might cause failures. They tested DuFP on images and text under different conditions and found it works better than other similar methods.
What this means in practice
- •For software testing teams: Identify high-risk inputs that likely cause failures in neural network software to prioritize testing efforts and reduce labeling costs.
- •For machine learning operations teams: Detect and handle ambiguous or unusual inputs in deployed models to improve reliability under standard, corrupted, and adversarial data conditions.
Authors
Haoran Li, Shihai Wang, Bin Liu, Jialuo Chen, Wenjing Zhu, Yu Liu, Tengfei Shi, Shudi Guo
Abstract
While Deep Neural Networks (DNNs) have achieved remarkable progress in cutting-edge domains, their inherent brittleness has become a growing concern. To ensure the reliability and safety of DNN-enabled software, DNN testing has emerged as an indispensable practice. Within this context, test input prioritization is essential for early fault detection and reducing labeling costs. However, it remains challenging to accurately identify failure-inducing inputs. Although decision ambiguity and distributional atypicality are two widely adopted perspectives for characterizing inter-class competition and intra-class typicality respectively, relying on either perspective in isolation inevitably introduces blind spots. In this paper, we propose DuFP (Dual perspective Feature space Prioritization), a KNN density-based test input prioritization approach for DNNs that jointly incorporates both inter-class and intra-class perspectives. The prioritization framework of DuFP is built upon class-conditional density estimation. Based on the estimation results, prediction correctness is characterized by an ambiguity score and an atypicality score, with the former reflecting decision ambiguity and the latter quantifying distributional atypicality. A hybrid uncertainty score is then constructed by integrating both scores to guide the final prioritization. We evaluate DuFP on prioritization and selection tasks across image and text datasets under clean, corrupted, and adversarial scenarios. Experimental results demonstrate that DuFP effectively and efficiently prioritizes fault-inducing inputs and outperforms state-of-the-art approaches.