Papers for

video content indexing teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Self supervised method improves time accuracy in video language models

SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models

Abstract: Video vision-language models (VLMs) have strong semantic understanding, yet remain unreliable at precise temporal grounding. Existing gains from explicitly aligning video frames with absolute time suggest that VLMs may already encode accurate temporal locations internally but lose precision when expressing them as timestamps. Our analysis across four VLMs and two grounding datasets confirms this hypothesis: attention intervals outperform predicted intervals on approximately 66% of valid examples. We term this discrepancy the temporal generation gap. Because precise temporal annotations are expensive, this finding motivates closing the gap to improve temporal grounding using only unlabeled video-query pairs. We propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal temporal attention as self-supervision for Group Relative Policy Optimization and applies a ground-truth-free criterion to select high-value samples. SAPO consistently improves temporal grounding across multiple datasets and models spanning weak to strong initial grounding ability. Relative mIoU gains reach 33.7% on Qwen2.5-VL-3B and 5.4% on the strong Time-R1-7B. The improvements transfer across datasets while preserving general video understanding.

Mon 19 OctComputer Vision and Pattern Recognition
The gist
It can be hard for computers to tell exactly when something happens in a video just by reading a description. The authors found that current video language models already know roughly the right time but struggle to give exact timestamps. They created a new way called SAPO that uses the model’s own internal attention clues to teach itself better timing, without needing extra labeled examples. This method improves the model’s ability to pinpoint when events occur in videos, and it works across different models and datasets.
Open → 2610.25923v1

Video object paths matched with text using detailed embeddings

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Abstract: Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and human-verified query--trajectory relations. Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.

Fri 16 OctComputer Vision and Pattern Recognition
The gist
Matching descriptions to specific moving objects in videos is tough because videos usually mix all objects into one summary. The authors created VTrajEmbed, which breaks videos down into individual object paths and links each path to matching text. They use two ways of encoding video frames to capture both movement and scene details, plus a special training method to avoid mistakes in matching. Their model performed better than previous ones at finding objects from text and vice versa. They also built a new test set to check how well models do at linking text to object movements.
Open → 2610.22842v1

GeoGAT improves video location guessing across city to continent levels

GeoGAT: Bidirectional Temporal Sampling Meets Hierarchical Graph Attention for Global Video Geo-localization

Abstract: Global video geo-localization aims to infer the geographic location of a video worldwide, evaluating performance across four geographic hierarchies: city, state/province, country, and continent. Existing methods typically employ one-way uniform sampling to process video frames and train independent classifiers for each hierarchy, which leads to the loss of key geographic cues and prediction conflicts between hierarchies, especially for complex multi-shot edited videos. To address these limitations, we propose GeoGAT, which integrates bidirectional temporal sampling with graph attention networks (GATs). Specifically, GeoGAT extracts forward and offset-reversed frame sequences to construct complementary spatiotemporal features. These fused features are then fed into a predefined geographical hierarchy graph, where GATs perform structure-aware message passing, while a dual-constraint mechanism prunes predictions to eliminate cross-hierarchy conflicts. We construct GeoGAT10k, comprising 9,720 multi-shot edited videos from 166 cities worldwide, specifically to benchmark generalization ability on complex video structures. Experimental results on CityGuessr68k and GeoGAT10k demonstrate that GeoGAT eliminates hierarchical conflicts entirely and achieves state-of-the-art performance across all four geographic hierarchies. On CityGuessr68k, GeoGAT outperforms the strongest baseline, evaluated under both classification and retrieval protocols, by 2.6 percentage points at the city level. On the more challenging GeoGAT10k with multi-shot edited videos, the accuracy improvement exceeds 24 percentage points, validating strong generalization to complex real-world scenarios.

Wed 30 SeptComputer Vision and Pattern Recognition
The gist
Figuring out where a video was taken anywhere in the world is tough, especially when videos jump between different scenes. The authors created GeoGAT, a system that looks at videos both forwards and backwards to get more clues about location. It then uses a smart graph method to understand how cities, states, countries, and continents relate, making more consistent and accurate guesses. They also made a new test set of tricky videos and showed their system works better than previous ones, especially on complex videos.
Open → 2609.39128v1

Audio visual models mistakenly match speech to faces by position

Uncovering Ordinal-Matching Bias in Audio-Visual LLMs

Abstract: This work aims to improve how audio-visual large language models (AVLLMs) associate speech with the correct visible speaker in multi-speaker scenes. We find that current AVLLMs frequently fail at this task, and analyze the nature of these failures. To this end, we construct a synthetic diagnostic dataset in which multiple visible speakers each utter a single word. Analysis on this corpus reveals a consistent error pattern across three recent open-source AVLLMs: models attribute utterances by simply matching the order of spoken sentences with the left-to-right, top-to-bottom arrangement of visible faces, rather than relying on audio-visual cues such as lip synchronization. We term this behavior \emph{ordinal-matching bias}. We further show that this bias can be substantially mitigated through a simple remedy, Ordinal-Decoupled Fine-Tuning (OD-FT), in which models are fine-tuned on synthetic videos where spatial positions of speakers and speaking order are independently randomized. Despite using only 400 synthetic training videos, OD-FT not only suppresses ordinal-matching bias but also improves audio-visual understanding on real-world videos, yielding average gains of 8.27\% for Qwen2.5-Omni and 2.57\% for video-SALMONN2+ across three audio-visual benchmarks.

Mon 28 SeptComputer Vision and Pattern RecognitionSound
The gist
Audio-visual large language models often try to match spoken words to faces in videos, but the authors found they usually just pair speech to the face's position on screen instead of using real audio and visual cues. They call this mistake ordinal-matching bias. To fix this, the authors created a new training method that randomizes speaker positions and speaking order during fine-tuning, which helps the models learn to associate speech and faces correctly. This improvement makes the models better at understanding who’s speaking in real videos.
Open → 2609.34223v1

Dual-side enhancements improve video segment retrieval without training

DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding

Abstract: Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query embedding to the video at test time, without finetuning the backbone or calling external large language models. Extensive experiments on three standard and two OOD benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.

Tue 8 SeptComputer Vision and Pattern Recognition
The gist
Finding the right part of a long video that matches a text description is hard and usually needs a lot of training. The paper introduces a new method that doesn’t require any extra training but still gets better at this task. They improve how videos are compared by looking at both single frames and groups of frames, and they also tweak the text description to better fit the video’s content. This approach works well even when the videos are different from the training examples used by other methods.
Open → 2609.08850v1