Context-Aware Interpretable Representations for Retrieval and Graph Convolutional Network Classification
Machine LearningComputer Vision and Pattern RecognitionInformation Retrieval
Summary
The authors address two problems in how computers understand images: existing methods don’t fully capture the true shape of the data (called the Geometric Gap) and their learned features are hard for humans to interpret (the Interpretability Gap). They propose a new technique combining Manifold Learning, which studies data shapes, with Rank-based Graph Embeddings that make the features easier to explain. Their method produces simpler, understandable representations without losing accuracy in tasks like finding similar images or classifying with limited labels. Experiments show that their approach balances being interpretable, compact, and effective.
Authors
Thiago César Castilho Almeida, Gustavo Rosseto Letício, Vinicius Atsushi Sato Kawai, Daniel Carlos Guimarães Pedronette
Abstract
The advances in visual information modeling and representation during the last decades are remarkable, mainly supported by Convolutional Neural Networks, Transformer-based, and Foundation Models. Despite this progress, critical challenges regarding the nature of similarity assessment and model transparency have been neglected. A primary concern is the Geometric Gap, where traditional pairwise measures fail to capture the intrinsic geometry of the dataset manifold. Furthermore, the Interpretability Gap persists, as representations often lack alignment with human cognition. Therefore, how to provide interpretability to representations while maintaining low dimensionality and high effectiveness in downstream tasks remains an open challenge. In this paper, we propose a novel unsupervised framework that integrates Manifold Learning strategies with Rank-based Interpretable Graph Embeddings. Our approach effectively bridges these gaps by first characterizing the contextual information of the dataset through manifold analysis and subsequently generating sparse, self-explainable embeddings. The proposed approach employs a flexible formulation, allowing different Manifold Learning and Representation Learning strategies. Extensive experimental evaluation across diverse datasets and features demonstrates that our Context-Aware representations not only provide intrinsic interpretability and dimensionality reduction but also maintain or enhance effectiveness in downstream tasks, specifically in image retrieval and semi-supervised classification using Graph Convolutional Networks (GCNs).