Multimodal retrieval improves by weighting inputs based on queries
Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval
Computer Vision and Pattern Recognition
Summary
Searching across videos, sounds, subtitles, and text together often doesn’t work better than just using one type alone because all types are treated equally. The authors created a method called SCALAR that decides which types of information are more important for each search and combines them accordingly. This approach helps the system find the right videos more often, even if some types of information are missing. They tested this method on several datasets and found it usually improved accuracy compared to previous ways that treated all inputs equally.
What this means in practice
- •For video search developers: Improve video search accuracy by weighting different data types like audio and subtitles based on the user query to get better results.
- •For multimedia recommendation teams: Enhance recommendation systems by combining multiple content types adaptively, even when some data types are missing or noisy.
Authors
Ambuj Mehrish, Anindya Nag, Sebastiano Vascon
Abstract
Multimodal retrieval integrates video, audio, subtitles, and text; however, recent geometric aggregators, such as Gramian volumes, hyperbolic volumes, and spectral objectives, treat all modalities symmetrically. Under a unified evaluation protocol, their joint scores frequently lag behind the strongest single-modality pathway by 1.9 to 27.6 R@1. Controlled analyses attribute this outcome to uniform modality influence. This work introduces Spherical Centroid Aggregation with Learned Adaptive Relevance (SCALAR), a query-conditioned aggregator that assigns relevance-based weights to each available modality before computing a spherical centroid. SCALAR accommodates arbitrary modality subsets and is trained on masked, reduced-arity views using rank-8 LoRA adapters. Across five benchmarks, SCALAR achieves positive aggregation gain on four, reaching +4.0 R@1, while none of the evaluated prior aggregators is positive on more than one. A uniform-weight ablation reproduces the degradation observed with symmetric aggregation. With only 4.8 million trainable parameters, SCALAR attains the highest text-to-video R@1 on three and performs within seed variation of the best result on a fourth. Under test-time modality dropout, SCALAR's representation-stage score surpasses the released GRAM checkpoint at every evaluated masking rate and benchmark by 3.2 to 10.9 R@1. Finally, as modalities are removed, rerankers trained exclusively on complete modality sets increasingly converge toward their video-only pathways, diminishing these representation-level gains and underscoring a limitation of standard two-stage retrieval pipelines.