Papers for

multimedia data platform teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Hypergraph module boosts video to text retrieval accuracy by up to eight points

Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval

Abstract: Volume-based multimodal retrieval jointly scores a text query with a candidate's video, audio, and subtitle embeddings. While this approach captures higher-order within-candidate alignment, the score remains candidate-local, and semantically related training samples primarily serve as contrastive negatives. This work introduces Hypergraph-Regularized Gramian Volumes (HyVol), a training-time module that incorporates these semantic relations prior to evaluating the original volume loss. Document hyperedges connect the observed modalities of each candidate, whereas semantic hyperedges link candidates whose detached captions are mutual top-k neighbors. A shallow gated hyper-graph network applies residual corrections to the modality embeddings. Presence masks exclude unavailable streams from message passing, and identity padding preserves the determinant of the observed Gram submatrix without feature imputation. As refinement operates on embeddings rather than scores, the same construction applies to both Gram and HyperGram. We remove the hypergraph after training, leaving the backbone-only architecture, original scoring function, and retrieval cost unchanged. We train both backbones on a 150K-clip subset of VAST-27M and evaluate zero-shot performance on six benchmarks. Under the paired protocol, HyVol improves R@1 across all five retrieval benchmarks, with video-to-text gains reaching +8.3 on MSR-VTT and +7.6 on VATEX. Under missing-modality masking, the V2T margin remains positive in all experimental settings, although the T2V margin becomes slightly negative in four.

Mon 14 SeptComputer Vision and Pattern Recognition
The gist
Multimodal retrieval means matching text queries with videos, their audio, and subtitles. Current methods mostly focus on matching each candidate's own data without using the relationships between different videos. The authors introduce a module called HyVol that uses a special graph to connect related videos during training, improving embedding quality. This module is removed after training, so it doesn't increase retrieval cost. Their approach improved retrieval scores on multiple benchmarks, especially for finding videos from text queries.
Open 2609.15320v1