Semantic grounding transforms test-time discovery into interpretable concepts
Relational Semantic Grounding for Test-Time Discovery
Artificial Intelligence
Summary
Sometimes machines see new things they weren’t trained on and only group them without naming them. The paper’s authors created a method that uses language models to describe known objects with meaningful features, then helps the system give names to new categories when they appear. This means the machine can explain new things it finds in words we understand, instead of just saying 'new group one' or 'new group two.' Their approach improves how well machines identify and explain things in the real world after they start working.
What this means in practice
- •For computer vision engineers: Enable systems to identify and semantically label new visual categories without retraining after deployment.
- •For ai application developers: Create AI that explains unknown objects encountered during operation using interpretable language concepts.
Authors
Tianle Liu, Gongbo Cheng, Chenggong Ni, Minming Gu, Tao Zhou, Fuyuan Hu, Fan Lyu
Abstract
Test-Time Discovery (TTD) aims to recognize known classes and detect novel ones from a continuous unlabeled test stream after deployment. Existing TTD methods mainly rely on visual clustering or prototype comparison, and can only assign anonymous labels to discovered classes, leaving them difficult to interpret in open-world applications. In this paper, we propose SEGA, a Semantic Grounding with Attributes framework for TTD. SEGA leverages large language models to generate attribute-rich descriptions for known classes and constructs an attribute-anchored visual-semantic space during training. At test time, we introduce Relational Semantic Grounding (RSG), which grounds unknown samples with similar and dissimilar known-class anchors, prompts the LLM to generate candidate concepts, and reranks them in the learned semantic space. The selected concept is further used for confidence-guided test-time semantic stabilization. By coupling attribute-level semantic alignment with relational concept grounding, SEGA transforms anonymous test-time clusters into interpretable semantic concepts. Experiments show that SEGA improves TTD baselines with the promise of semantic grounding for open-world.