Summary
Object detection systems that use multiple models or multiple images often combine their results to reduce mistakes. Common ways to check if these systems work well look only at the final combined answer, missing important details about how the different models agree or differ. The authors created two new scores that look at the agreement and disagreements among the versions before combining them, helping to understand and improve how trustworthy the combined system is. They tested these scores in a self-driving car simulator and showed that these scores help pick better combinations of models, choose how to combine their answers, and build larger systems step-by-step with confidence.
What this means in practice
- •For autonomous vehicle engineers: Improve reliability by selecting and combining object detection models based on complementary errors and agreement patterns.
- •For surveillance system developers: Design multi-camera or multi-algorithm setups with guidance on version selection and result voting strategies using new reliability measures.
Tested on simulated data.
Abstract
N-version object detection (OD) is an approach to diversifying detection results using multiple models or input frames and reducing detection errors by aggregating individual results. Diversity and consistency across multiple detection results are critical information for characterizing the reliability of possible configurations of N-version OD systems. However, existing performance metrics such as mAP and Accuracy fail to capture these factors, as they are defined solely on the final outcome after aggregation. To overcome this limitation, we propose two reliability metrics particularly defined for N-version OD, namely coverage of errors in OD (Cov_OD) and certainty of accurate prediction in OD (Cer_OD), which can be computed from individual detection results without relying on voting strategies. We empirically demonstrate the unique features of the proposed metrics through a case study of N-version OD for a vehicle in an autonomous-driving simulator. We show that the proposed metrics can be used for 1) selecting version combinations with complementary error characteristics, 2) choosing an effective voting strategy based on diversity and consistency profiles, and 3) guiding incremental construction of N-version OD systems via pairwise two-version analysis. These results highlight the importance of the metrics that can guide the design of reliable N-version OD applications.