Final layer probing reveals structured computations in vision language models
FLIP: Final Layer Inference-Time Probing for Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding how vision-language models make decisions is tricky because changes inside the model can have many unclear effects. The authors propose a method called FLIP, which gently changes the model’s last internal layer to see if the model uses visual information in an organized way for tasks like object detection and counting. By carefully analyzing how these changes affect performance, they identify a specific layer state that reliably supports meaningful task-related processing. This method helps confirm that some internal parts of the model truly do structured work, rather than just causing generic instability.
What this means in practice
- •For machine learning engineers: Validate the mechanistic role of specific internal layers in vision-language models to improve reliability of model interpretation.
- •For ai safety teams: Use reliable intervention tests at final layers of vision-language models to detect task-relevant computations versus generic output changes.
Authors
Drandreb Earl O. Juanico, Rowel O. Atienza
Abstract
We present FLIP, a final-layer inference-time probe for testing whether a logit-facing intervention site in an open-weight vision-language model (VLM) supports structured, task-linked computation rather than generic perturbation. Behavioral change under internal intervention is otherwise mechanistically ambiguous: it may reflect improved use of visual evidence, generic output instability, or outright degradation. FLIP applies elementwise flooring to the final normalized hidden state before logit computation, leaving parameters, prompts, and decoding unchanged. On a controlled detection/counting probe, sweeping intervention strength reveals three regions: negligible change, a bounded interior regime in which detection recall at IoU 0.50 ($R_{50}$) improves while tolerant counting error ($\mathcal{E}_{\mathrm{count}}$) falls, and over-suppression. We formalize a four-criterion probe-and-sweep protocol for disciplining the interpretation of intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and failure to reproduce the same positive regime on a performance-based negative control. The post-normalization state passed to the output head is the logit-facing instantiation of this test; under a non-targeted flooring sweep it satisfies the full protocol. Raw decoder-layer interventions, including the last-block output before final normalization, and the singleton-pair left/right control fail to reproduce the Final-site signature, while same-site operators and multiple VLMs replicate it. FLIP is therefore a validation step for intervention-based mechanistic interpretability, not a steering method.