Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address the challenge of using Graph Convolutional Networks (GCNs) for image classification, where it's hard to make graphs because images don't have natural connections like citation networks. They improve graph building by using Vision Language Models to create text descriptions of images and Large Language Models to measure how similar those descriptions are, helping to remove irrelevant connections between images. This method helps the GCN focus on more meaningful relationships, leading to better classification accuracy in some cases. Their approach shows that combining language understanding with image analysis can refine graphs for semi-supervised learning.
Graph Convolutional NetworksSemi-supervised learningImage classificationGraph constructionk-Nearest Neighbors (kNN)Vision Language ModelsLarge Language ModelsSemantic similarityFeature vectorsGraph pruning
Authors
Camila Piscioneri Magalhães, Lucas Pascotti Valem
Abstract
While the growing availability of image data has driven significant advances, labeling datasets remains costly and time-consuming. Therefore, semi-supervised approaches such as Graph Convolutional Networks (GCNs), which learn from both labeled and unlabeled data, have emerged as a promising solution. One of the primary challenges in applying GCNs to image classification is graph construction, since, unlike in citation networks or similar domains, images typically do not come with a predefined structural representation. For visual data, most studies construct graphs based on the similarity between feature vectors from pretrained deep learning backbones, typically by employing kNN or reciprocal kNN algorithms. Although Large Language Models (LLMs) have shown remarkable capability in capturing high-level semantics, their integration with GCNs for image classification remains underexplored. Aiming to fill this gap, our approach uses a Vision Language Model (VLM) to generate textual image descriptions, which are then processed by an LLM to estimate semantic similarity scores between connected images. These scores guide the pruning of edges in kNN and reciprocal kNN graphs, filtering out semantically irrelevant neighbors. Experimental results reveal that leveraging LLMs for graph refinement can improve classification accuracy, particularly for kNN graphs and some backbones. The source code is publicly available at http://gcnllm.lucasvalem.com.