AI summaryⓘ
The authors studied how to choose the best layer of a vision model to use in vision-language models like LLaVA, since the common choice of a fixed late layer often performs poorly across tasks. They tested two methods to predict useful layers without expensive trial-and-error: Visual Dataset Entropy (VDE), which measures information in visual features, and Gromov-Wasserstein (GW) distance, which measures alignment between vision and language parts. They found VDE works well before the multimodal projector and can guide layer selection from a small number of unlabeled samples, while GW is less useful for this purpose. This method helps reduce the effort needed to find good layers for different tasks without retraining or labels.
Vision-Language ModelsLLaVAVision BackboneVisual TokensVisual Dataset Entropy (VDE)Gromov-Wasserstein DistanceMultimodal ProjectorLayer SelectionRepresentation GeometryModel Evaluation
Authors
Ruchen Liu, Yi Yang, Yiming Xu, Michael Ying Yang, Monika Sester, Bodo Rosenhahn
Abstract
LLaVA-style Vision-Language Models (VLMs) pass visual tokens from a fixed late layer of the vision backbone, typically the penultimate one, to the language model. We first show that this hidden convention is fragile: across 2 VLMs and 7 image and video benchmarks, the default layer is sub-optimal in 13 of 14 model-task pairs, and the best layer shifts with both task and visual backbone. Finding that layer by exhaustive layer-wise inference is prohibitively expensive, and no better fixed default exists. We therefore ask whether layer usefulness can instead be predicted from representation geometry. We study matrix-based entropy, introduced for unimodal layer analysis, which we compute over sample-level visual embeddings as Visual Dataset Entropy (VDE); and Gromov-Wasserstein (GW) distance, introduced for encoder-level VLM model selection, which we repurpose as a layer-wise visual--language alignment signal. Transferring these to LLaVA-based models is not obvious a priori: the vision tower is frozen while the multimodal projector is trained, so we profile both sides of the projector. We find that VDE transfers, and GW does not. Computed from 100 unlabeled task samples without downstream inference, pre-projector VDE tracks layer-wise accuracy and its top-ranked layers cover the oracle best layer on every task for the SigLIP-based LLaVA-Video, while giving region-level guidance for the CLIP-based Video-LLaVA. Post-projector profiles show that the projector reshapes visual geometry but does not erase the performance-relevant trend, leaving $\mathrm{VDE}_{\mathrm{pre}}$ the stronger signal. GW instead flattens after projection and is best read as an alignment diagnostic rather than a selector. VDE thus offers an interpretable, training-free policy that narrows the visual-layer search to a handful of candidates for limited downstream verification.