Unified multimodal video search finds ordered events using evidence from video text speech
EVISTA: Unified Multimodal Video Search with Evidence-Aware Temporal Path Optimization
Information Retrieval
Summary
Finding specific events in long videos can be hard, especially when those events involve different kinds of clues like images, text on screen, or spoken words. The authors built a system called EVISTA that searches videos using all these clues together, rather than separately. To make sure the system finds multiple events in the right order, they created a method called TEMPO-P that looks for evidence across different types and picks a sequence of moments in the video that makes sense. Their method works well even when some events are only described by text or speech and not by visuals.
What this means in practice
- •For video platform developers: Improve search engines to find multiple ordered events across video content using visual, text, and speech evidence combined.
- •For media monitoring teams: Automatically locate and order events of interest in large video collections for research and analysis using multimodal clues.
Authors
Minh-Huy Tran, Van-Thinh Vo, Minh-Khoi Nguyen, Duy-Tan Nguyen, Anh-Quan Nguyen-Tran, Loi Nguyen Khanh
Abstract
In the era of big data, precise event retrieval from large video collections has become increasingly important and challenging. In particular, locating multiple events from the same video in their temporal order is difficult when different events are described by different types of evidence, such as visual content, on-screen text, and spoken words. We develop EVISTA, a unified multimodal video retrieval system for the Ho Chi Minh City AI Challenge 2026, which integrates visual-semantic, OCR, ASR, object, and metadata search over a shared keyframe index, and propose TEMPO-P, an evidence-aware temporal path optimization method for ordered multi-event retrieval. Building EVISTA for this task reveals two limitations of independent event retrieval: a relevant video may be missed when one event is weakly retrieved, and independently retrieved event candidates may be temporally inconsistent or unable to represent events defined primarily by text or speech. TEMPO-P addresses these limitations by aggregating candidate videos across events and modalities, jointly scoring keyframes using visual, OCR, and ASR evidence, and selecting a compact temporally ordered path within each video through dynamic programming. On five organizer queries and 36 additional multi-event queries, TEMPO-P matches visual-only temporal alignment on visually described events (R@1 0.42 for both) and retrieves queries containing text-defined events, which visual-only alignment cannot represent (R@1 0.38). TEMPO-P is integrated into EVISTA as its ordered-event retrieval mode.