Semantic mismapping distorts accuracy in category discovery methods
Semantic Mismapping in Generalized Category Discovery: Evaluation and Mitigation
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
When computers try to recognize old categories and discover new ones at the same time, they use a matching process to compare predictions with labels. The authors found that this process can mistakenly swap the known categories, making the new category recognition look better than it really is. They call this problem semantic mismapping. To fix it, they designed a better way to evaluate and train the system that respects the known categories and reduces errors, leading to more accurate results.
What this means in practice
- •For machine learning engineers: Improve training and evaluation of models that identify known and unknown categories in unlabeled data.
- •For computer vision teams: Enhance accuracy in recognizing existing and new object classes simultaneously during image analysis.
Authors
Zitai Wang, Qianqian Xu, Cong Hua, Xi Chen, Zhiyong Yang, Xiaochun Cao, Qingming Huang
Abstract
Generalized Category Discovery (GCD) aims to perform old-category recognition and novel-category discovery in the same unlabeled set. Early methods address both objectives through clustering and evaluate the exchangeable predicted indices with global Hungarian (GH) matching. Recent methods adopt trainable parametric heads, using supervision for old-category recognition and discovery-oriented objectives for new categories. In this way, the output space becomes partially exchangeable: old indices are semantically anchored, while new indices remain exchangeable. However, these methods still apply global matching, which can reassign anchored old indices to other categories and hide some prediction errors. Empirically, we are surprised to observe that this reassignment can induce an overestimation of new-category accuracy by up to 25.63 points. We term this phenomenon semantic mismapping and further derive a permutation-cycle condition that characterizes when semantic mismapping can increase the global matching score. To fix this issue, we propose Semantics-Preserving Evaluation (SPE), which preserves the supervised semantics of old indices and restricts Hungarian matching to the new region. Furthermore, we propose Mismapping-Mitigating Optimization (MMO) to alleviate the prediction errors that can give rise to semantic mismapping, combining a region-balance loss (RBL) for old/new routing with a novel-diversity loss (NDL) that encourages orthogonality among new prototypes. Theoretically, we show that minimizing the two losses can tighten an upper bound on spurious accuracy gains from semantic mismapping, with controllable residual terms. Empirical results under both GH and SPE demonstrate the effectiveness of our proposed method.