Vision language model clusters faces online in long videos
VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos
Computer Vision and Pattern RecognitionMultimedia
Summary
Clustering faces from long videos is hard because faces change over time with different lighting, angles, or costumes. The authors present VISTA, a system that uses both images and language information to group faces as they appear in a video stream without needing to see all the video at once. This approach helps keep the identity groups accurate even when faces look very different over time. Tests on episodes from TV shows show VISTA performs better than previous methods that only use visual features.
What this means in practice
- •For video surveillance teams: Improve real-time face grouping in long camera feeds despite changes in appearance or lighting.
- •For film and media archivists: Automatically organize and tag characters in long-form video content progressively as footage is processed.
Authors
Jiashu Liao, Joanne Garde-Hansen, Sanjay Sharma, Tanaya Guha
Abstract
Clustering faces by identity in long, continuous video streams is a challenging task that serves as a building block for human-centric video analysis. The majority of existing face clustering models are offline, i.e., they assume the entire data to be available altogether. In many practical scenarios, however, the entire data is either unavailable or poses computational challenges to be processed in its entirety. A handful of online methods that exist rely on geometric visual features and simple thresholding techniques, which fail when the faces exhibit significant appearance variations in long, unconstrained videos. To this end, we present VISTA, the first vision language framework for online face clustering in long videos. VISTA leverages semantic reasoning in a controlled, uncertainty aware fashion to maintain both high cluster purity and long range identity continuity. This makes it robust to the variations in illumination, pose, and costume changes that the faces and characters exhibit in long real world videos. Extensive experiments on the VPCD benchmark dataset (with 15 full episodes from The Big Bang Theory [2007-2019], Buffy the Vampire Slayer [1997-2003], and Sherlock [2010-2017]) demonstrate that VISTA achieves a robust precision and recall balance, consistently improving over the strongest matched online unsupervised baseline (COFC) under identical detector, embeddings, and evaluation protocol. Code and supplementary material: https://github.com/SkyJiashu/VISTA.