Face voice matching struggles with language and gender shifts
Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan
Computer Vision and Pattern Recognition
Summary
Some computer models try to match a person's voice to their face, but they often rely on clues like language or whether the person is male or female rather than unique features of the voice. This means these models don’t work as well when the speaker speaks a new language or when faces to choose from share the same gender. The authors created a new challenge, called FLAG 2027, to test and improve how well these models can match voices to faces across different languages and genders. They also provide data, baseline models, and tools for others to build better systems.
What this means in practice
- •For security system developers: Improve identity verification systems that match voices to faces even when speakers use different languages or share the same gender.$Commercial implications: Enables selling more reliable biometric authentication products that handle multilingual and gender-diverse users.
- •For multimedia platform engineers: Enhance cross modal retrieval tools on platforms by accurately linking user voices to their faces despite language or gender variations.
Authors
Marta Moscati, Swapnil Khandoker, Muhammad Saad Saeed, Shah Nawaz, Fatima Noor, Rohan Kumar Das, Mubashir Noman, Junaid Mir, Muhammad Haroon Yousaf, Khalid Malik, Markus Schedl
Abstract
Face--voice association models may rely on language or gender cues in the voice rather than on speaker-specific voice characteristics, which can lead to a performance deterioration when the model has to identify a multilingual speaker or distinguis same-gender speakers. To investigate these issues, we introduce the Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge. The challenge formulates face--voice association as a cross-modal verification task: given a voice, identify the speaker's face from a ``gallery'' of faces consisting of the speaker's face and a set of negative samples. Models are evaluated on identities not present in the training data (``unseen'') and both for languages present or absent from the training data (``heard'' and ``unheard''). Two evaluation settings are used to test models' reliance on gender: a standard, unconstrained and a gender-constrained one, where the latter uses a same-gender gallery. The performance of existing, baseline models in these settings reveals that models performance degrades under language shifts and in gender-constrained settings, highlighting the need to foster the development of models that capture identity-specific aspects beyond language and gender. The challenge provides a benchmark dataset, pretrained baseline models, and an evaluation framework to advance face--voice association.