Neural network open set recognition improved by modular two stage method
Circumventing Magnitude Collapse via Modular Two-Stage Open-Set Recognition
Computer Vision and Pattern Recognition
Summary
Open-set recognition involves teaching computers to recognize familiar things while ignoring unfamiliar ones, which is important for real-world use. The authors found a problem called magnitude collapse where certain learning methods perform well with small data but fail badly with large data, losing important details. They studied how different methods interact and showed how using a two-stage approach with special adjustments can avoid these failures and improve recognition. They also found that tests on small data do not predict how well these methods work on larger, real-world data.
What this means in practice
- •For machine learning engineers: Implement two-stage open-set recognition pipelines to improve reliability when deploying models on large and diverse datasets.
- •For security system developers: Use magnitude-aware postprocessing to better detect unknown or unexpected inputs in real-time AI-based security systems.
Authors
Laurin van den Bergh, Halil Bisgin, Andres Palechor, Mike Suter, Manuel Günther
Abstract
Open-set recognition (OSR) requires neural networks to classify known classes while rejecting unknown samples, which is critical for real-world deployment. So far, OSR research studied and developed representation learning and postprocessing methods independently and their interactions remain unexplored. In this paper, we reconsider OSR as a modular two-stage pipeline and present the first systematic study of interactions between representation learning and postprocessing methods across dataset scales and usage of known unknown data. First, we discover a failure mode we term \emph{magnitude collapse}, where representation learning methods that utilize known unknown data can suffer performance degradation at large scale and irreversibly destroy discriminative information, despite excelling at small scale. Second, we study the modular interactions and reveal when they can be leveraged for modular performance gains via two-stage processing and where interaction effects amplify performance degradation due to magnitude collapse. Third, we show how these findings can be used to achieve state-of-the-art performance with a simple $K+1$ class classifier baseline and magnitude-aware postprocessing while circumventing magnitude collapse. Finally, our results demonstrate that small-scale evaluations with known unknown data are not predictive of large-scale performance, invalidating current best practices in OSR research. Code available at https://github.com/TwoDigitsOneNumber/Two-Stage-OSR.