Hypergraph module boosts video to text retrieval accuracy by up to eight points
Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval
Computer Vision and Pattern Recognition
Summary
Multimodal retrieval means matching text queries with videos, their audio, and subtitles. Current methods mostly focus on matching each candidate's own data without using the relationships between different videos. The authors introduce a module called HyVol that uses a special graph to connect related videos during training, improving embedding quality. This module is removed after training, so it doesn't increase retrieval cost. Their approach improved retrieval scores on multiple benchmarks, especially for finding videos from text queries.
What this means in practice
- •For video search engineers: Improve video search engines by enhancing text queries to video matching accuracy using hypergraph-based embedding refinement during training.
- •For multimedia data platform teams: Upgrade indexing systems to better handle multiple data types like video, audio, and subtitles simultaneously by applying hypergraph regularization techniques to embeddings.
Authors
Anindya Nag, Ambuj Mehrish, Sebastiano Vascon
Abstract
Volume-based multimodal retrieval jointly scores a text query with a candidate's video, audio, and subtitle embeddings. While this approach captures higher-order within-candidate alignment, the score remains candidate-local, and semantically related training samples primarily serve as contrastive negatives. This work introduces Hypergraph-Regularized Gramian Volumes (HyVol), a training-time module that incorporates these semantic relations prior to evaluating the original volume loss. Document hyperedges connect the observed modalities of each candidate, whereas semantic hyperedges link candidates whose detached captions are mutual top-k neighbors. A shallow gated hyper-graph network applies residual corrections to the modality embeddings. Presence masks exclude unavailable streams from message passing, and identity padding preserves the determinant of the observed Gram submatrix without feature imputation. As refinement operates on embeddings rather than scores, the same construction applies to both Gram and HyperGram. We remove the hypergraph after training, leaving the backbone-only architecture, original scoring function, and retrieval cost unchanged. We train both backbones on a 150K-clip subset of VAST-27M and evaluate zero-shot performance on six benchmarks. Under the paired protocol, HyVol improves R@1 across all five retrieval benchmarks, with video-to-text gains reaching +8.3 on MSR-VTT and +7.6 on VATEX. Under missing-modality masking, the V2T margin remains positive in all experimental settings, although the T2V margin becomes slightly negative in four.