AI summaryⓘ
The authors developed SCOUT, a new tool that helps find and change meaningful features inside face recognition templates, which are compact digital representations of a person's face. Unlike previous methods that rely on editing images or manually labeling features, SCOUT automatically discovers and adjusts these features directly in the template space using natural language descriptions. This allows for precise and flexible changes without needing to reprocess images, and the edited templates still work well for recognizing identities. The authors tested SCOUT with different face recognition models and showed it can control features beyond usual facial attributes while preserving identity.
Face recognition templatesSemantic editingMechanistic interpretabilitySparse representationsNatural language descriptionsCNNViTSwin TransformerTemplate inversionIdentity matching
Authors
Leon Todorov, Peter Rot, Peter Peer, Vitomir Štruc, Klemen Grm
Abstract
Face recognition templates are compact identity representations, yet they also encode rich semantic information about facial appearance. Prior work has shown that templates can be inverted to images or indirectly manipulated through image-editing pipelines, but direct semantic editing in template space remains largely unexplored. Existing interpretability methods for face recognition often rely on manual neuron inspection or predefined attribute labels, limiting scalability and semantic flexibility. To address this gap, we propose SCOUT (Semantic Concept Discovery for Open-VocabUlary Editing of Face Recognition Templates), an end-to-end framework for discovering and directly manipulating semantic concepts in face recognition templates using mechanistic interpretability. SCOUT learns sparse template representations, generates semantic hypotheses for latent features from natural-language descriptions, and validates their stability. The resulting features act as controllable semantic directions for direct editing, avoiding costly edit--re-encode pipelines. Experiments with face recognition models using CNN, ViT, and Swin backbones show that SCOUT discovers interpretable concepts beyond standard attribute labels and enables controllable, identity-aware template manipulation with negligible impact on identity matching. We further show that edited templates can subsequently be decoded with independent inversion models for visualization and evaluation.