Multimodal video retrieval improves when fusion order matches query details
When Fusion Order Matters: Insights on Progressive Multimodal Representation Learning for Cross-modal Retrieval
Artificial IntelligenceInformation Retrieval
Summary
Finding the best way to combine video, sound, and speech information for searching videos can be tricky. The authors tested different orders of mixing these types of information to see which work best depending on the search query. They found that adding the information type matching the search last often helps retrieval. Their approach helps choose the right order for combining data to improve search accuracy.
What this means in practice
- •For video platform developers: Optimize video search systems by tuning the order of combining visual, audio, and speech features based on query type for better relevance.
- •For multimedia indexing teams: Improve indexing accuracy for multimedia archives by selecting fusion schedules that balance missing data and query focus.
Authors
Haoxiang Zhao, Bin Wang, Yaofei Wang, Xiao Lan, Yunfeng Diao, Xu Wang
Abstract
Multimodal video retrieval combines visual, acoustic, and speech information, yet how their fusion order should depend on the data and the retrieval query remains unclear. We introduce Unified Progressive Representation Learning for Order Analysis, or UniPRO, to investigate this question through configurable fusion schedules with matched model components and training settings. Starting from text-to-video retrieval, we find that the tested one-by-one schedules outperform stacked fusion, while changing the order within one-by-one fusion also changes performance. To guide order selection, we examine two complementary considerations: learning difficulty under unequal modality availability and preservation of information relevant to the query. Complete-modality controls show that missing inputs alone cannot explain order sensitivity. At R@1, text and audio retrieval favor schedules that introduce the modality matching the query last. Controlled speech missingness during training and testing reverses the preferred order in both R@1 and SumR, with different crossover points. These findings motivate balancing the two considerations when selecting a fusion path. We then use them to predict a visual-last schedule for image queries, which achieves 97.9 R@1, the highest among the evaluated orders on MSR-VTT-9k. Main experiments on four text-to-video benchmarks establish the effectiveness of UniPRO, with component ablations assessing its supporting design. Together, the results identify fusion order as a task-dependent choice that should account jointly for modality availability and the retrieval query.