Vision language models share geometric patterns in low level tasks
Geometric Similarity in VLM Low-Level Vision Representations
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models can understand images and language but it's unclear how they organize information for detailed image fixes like sharpening or denoising. The authors studied how these models represent 24 different low-level image tasks and found that despite architectural differences, they share some common geometric patterns in the way they represent pixel information. This insight can help improve how these models adapt to various image restoration tasks and highlight where they still struggle. Their method, GeoSim, offers a way to better interpret and transfer knowledge between different vision-language models.
What this means in practice
- •For computer vision engineers: Develop unified image restoration tools that efficiently adapt across multiple low-level vision tasks using insights about shared model representations.
- •For machine learning platform teams: Diagnose and improve transfer learning capabilities for vision-language models by analyzing geometric similarities between tasks and models.
Authors
Shao-Jun Xia, Huixin Zhang, Zhen Lei, Anlan Sun, Yuner Zhang, Xiaoyang Chen
Abstract
Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers (DiTs). Yet, adapting them efficiently for all-in-one low-level image restoration remains a challenge. Crucially, the field lacks an understanding of how VLMs organize hidden-layer representations and whether these structurally distinct paradigms share a common geometric organization for pixel-level perception. Such shared organization is a prerequisite for building highly transferable, unified restoration VLMs and adapters. In this paper, we systematically investigate representational similarity across 24 low-level tasks spanning 5 categories. We propose GeoSim, a unified four-level framework that analyzes task-conditioned representations from global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. Our formulation applies to the analysis of hidden states in AR models and feature maps in DiTs across same- and cross-task/model settings. Our results reveal the organizing principles of low-level visual representations while exposing their limits in cross-task and cross-model agreement. Ultimately, GeoSim provides an interpretability lens for probing latent transferability in low-level vision and diagnosing model limitations in task- or model-specific scenarios.