Theory clarifies environment-aware out-of-distribution detection limits
A Theoretical Analysis of Environment-Aware Out-of-Distribution Detection for Vision-Language Models
Machine Learning
Summary
Vision-language AI models need to tell when new images do not belong to any known category, especially as they encounter changing environments over time. The authors developed a theoretical approach that measures how well images and text descriptions from different environments match, improving the detection of out-of-distribution inputs. Their theory explains when using earlier environments helps or harms detection and provides guarantees on false positives. They also show when it is impossible to reliably reject certain unknown inputs based on the model's current knowledge.
What this means in practice
- •For machine learning engineers: Know when including past environment data will improve or worsen detection of new unknown inputs in vision-language systems.
- •For system designers: Understand theoretical limits of multi-environment out-of-distribution detection to guide model design for reliable unknown input rejection.
A theory result. No direct application yet.
Authors
Salimeh Sekeh, Mary Wisell
Abstract
Continual vision-language models (VLMs) encounter a stream of environments and must also reject inputs that do not belong to the environment at hand. Class information from earlier environments can aid out-of-distribution (OOD) detection, yet existing detectors score each environment in isolation, and when previous environments help or hurt is theoretically unclear. We develop a theory of environment-aware OOD detection built on contribution scores, pairwise cosine expectations measuring how the image and text features of one environment align with those of another. We propose a cross-environment maximum-concept-matching (MCM) score with two terms: a primary term matching the test image against the current class texts, and a secondary term matching it against the class texts of a previous environment, weighted by the intra-modal alignment between the two environments. We prove that, under a positive environment transition and a bounded class-matching margin for OOD inputs, this score attains a false-positive rate no larger than that of the standard MCM detector at an adjusted threshold. We then show that the score gap of an MCM detector between two environments equals the self-contribution of the in-distribution environment minus its cross-modal contribution into the OOD environment, which separates soft OOD environments, with explicit AUROC, true-positive, and false-positive guarantees, from hard ones, which no threshold that retains the in-distribution data can reliably reject. For cascaded detection, in which the text prototypes of all seen environments are pooled, the false-positive rate on every past environment is non-decreasing in time, and we characterize when detection provably fails. A contribution balance factor applied to the gradients of the classification loss and of a smoothed false-positive rate completes the framework.