Optimal transport measures disagreement in object detection labels
Quantifying Annotation Agreement in Object Detection with Optimal Transport
Computer Vision and Pattern Recognition
Summary
Validating machine learning models relies on having correct labels in test data, but sometimes people disagree when labeling objects in images. The authors studied a new way to measure how much annotations disagree using a mathematical tool called optimal transport. Unlike traditional methods that treat label matches as all-or-nothing, this approach scores partial overlaps and differences more precisely. This helps better understand ambiguity in labels, especially when objects are small or unclear, improving trust in model validation.
What this means in practice
- •For machine learning engineers: Improve evaluation of object detection models by measuring nuanced differences in annotated test sets with optimal transport.
- •For data labeling teams: Identify and quantify ambiguous or inconsistent labels in object detection tasks to guide better annotation practices.
Authors
Kirill Prokhorov, Alexandr A. Kalinin
Abstract
Reliable validation of machine learning models is a crucial step in designing robust and trustworthy systems. Yet validation relies on annotated test sets treated as ground truth, even though annotations can be noisy or ambiguous. In object detection, sources may disagree on object presence, box location, or class label. Standard agreement metrics usually binarize box overlap before computing a summary score, obscuring the degree and structure of disagreement between annotation sets. Here, we study an Optimal Transport (OT)-based metric for measuring agreement between object detection annotations. Instead of binarizing bounding box overlap, this metric assigns graded costs to matched and unmatched labeled boxes. Through theoretical analysis and experiments on real and synthetically perturbed annotations, we show that OT captures graded changes in agreement and distinguishes annotation pairs that thresholded metrics score identically. Our results suggest that OT provides a richer measure of annotation agreement in object detection, especially for domains with many small or ambiguous objects. Such agreement estimates can support more reliable validation when labels are noisy or conflicting.