Papers for

film and media archivists

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Vision language model clusters faces online in long videos

VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos

Abstract: Clustering faces by identity in long, continuous video streams is a challenging task that serves as a building block for human-centric video analysis. The majority of existing face clustering models are offline, i.e., they assume the entire data to be available altogether. In many practical scenarios, however, the entire data is either unavailable or poses computational challenges to be processed in its entirety. A handful of online methods that exist rely on geometric visual features and simple thresholding techniques, which fail when the faces exhibit significant appearance variations in long, unconstrained videos. To this end, we present VISTA, the first vision language framework for online face clustering in long videos. VISTA leverages semantic reasoning in a controlled, uncertainty aware fashion to maintain both high cluster purity and long range identity continuity. This makes it robust to the variations in illumination, pose, and costume changes that the faces and characters exhibit in long real world videos. Extensive experiments on the VPCD benchmark dataset (with 15 full episodes from The Big Bang Theory [2007-2019], Buffy the Vampire Slayer [1997-2003], and Sherlock [2010-2017]) demonstrate that VISTA achieves a robust precision and recall balance, consistently improving over the strongest matched online unsupervised baseline (COFC) under identical detector, embeddings, and evaluation protocol. Code and supplementary material: https://github.com/SkyJiashu/VISTA.

Tue 3 NovComputer Vision and Pattern RecognitionMultimedia
The gist
Clustering faces from long videos is hard because faces change over time with different lighting, angles, or costumes. The authors present VISTA, a system that uses both images and language information to group faces as they appear in a video stream without needing to see all the video at once. This approach helps keep the identity groups accurate even when faces look very different over time. Tests on episodes from TV shows show VISTA performs better than previous methods that only use visual features.
Open → 2611.04932v1