Vision language models face challenges explaining face recognition results
Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Face recognition technology needs to explain why two faces match or not, especially for important decisions like forensic cases. This paper proposes a new way to test not just how well these models identify faces, but also how clear and truthful their explanations are. The authors created methods to check if explanations focus on stable facial features and if they accurately describe the images without making things up. They tested several open models and found current explanations often fall short, showing the importance of measuring explanation quality.
What this means in practice
- •For forensic technology developers: Evaluate and improve forensic face recognition tools by incorporating measures of explanation clarity and trustworthiness.
- •For security system engineers: Enhance face verification systems by benchmarking both identification accuracy and explanation reliability to increase user confidence.
Authors
Laurent Colbois, Sébastien Marcel
Abstract
Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations alongside similarity scores. This capability is considered appealing for face comparisons in forensic contexts, which require decisions to be transparent and auditable. However, existing evaluations of VLMs for that use case focus mostly on recognition accuracy, while the validity of generated explanations remains unquantified. In this work, we introduce a benchmarking framework for VLM-based face recognition that treats explanation quality as a core evaluation axis. We propose two criteria that explanations should satisfy: relevance, i.e., reliance on identity-stable facial features; and faithfulness, i.e., alignment with the visible image content without hallucinated features. We jointly develop a methodology enabling the quantification of relevance and faithfulness of evaluated models, based on constraining model outputs to a structured explanation format that supports automated querying and auditing. Using this framework, we benchmark several families of open-weight VLMs, jointly evaluating face verification accuracy and explanation quality. Our results highlight remaining shortcomings of produced explanations, and emphasize the need for such explanation quality metrics to get a complete picture of model performance. The proposed benchmark and open-source evaluation harness provide a foundation for proper benchmarking and future fine-tuning of explainable face recognition systems.