Film character identification from dialogue and video like a human
As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does
Computer Vision and Pattern Recognition
Summary
Movies introduce characters bit by bit, and figuring out who is who is tricky without extra information like cast lists. The authors study how to identify key characters and their names just from the movie's video and dialogue, similar to how humans do it. They look at cues like how people speak, where they look, and which shots follow one another. The authors created a dataset and a model that can name characters as the film unfolds, and compared it to existing language-based models. This approach helps computers understand movies in a more human-like way by recognizing characters from the film itself.
What this means in practice
- •For video streaming platforms: Automate tagging and indexing of key movie characters based on on-screen dialogue and video alone.$Commercial implications: Enables smarter content navigation and recommendation features by identifying characters without extra metadata.
- •For film post-production teams: Assist editors by automatically recognizing characters during editing without relying on external cast or actor data.
Authors
Zhongrui Gui, Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman
Abstract
Films introduce characters incrementally as they progress. We study the problem of identifying principal characters and assigning their names directly from the movie itself, without using external cast lists, actor photographs, or pre-built character banks. We introduce the character naming task: given a name mentioned in dialogue and the associated video context, the model must determine which visible character bears that name. This task is challenging because name resolution often depends on cinematic cues such as shot-reverse-shot structure, self-introduction, direct address, third-party reference, gaze, and temporal continuity. We provide a dataset and annotations for evaluating character naming and first character appearances, develop a video-and-dialogue model for the task, and compare it with proprietary multimodal large language models. We further introduce a progressive framework for building a character bank as the film unfolds. Together, these contributions support online, human-like movie understanding in which character identities are inferred from the film itself.