Multimodal embeddings balance detail and efficiency with residual compression

ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Multimodal large language models struggle to represent images and text together efficiently—either they use one summary vector that misses details, or many detailed vectors that take a lot of space. The authors designed ResComEmb, a new method that compresses redundant parts of multimodal inputs while keeping important details by looking at different detail levels. This makes embeddings smaller and faster to use without losing quality. They tested it on image document tasks and found it outperforms previous methods while using fewer resources.

What this means in practice

  • For visual search developers: Improve visual document retrieval systems by reducing embedding size while maintaining or increasing accuracy using ResComEmb.
  • For multimodal ai engineers: Build efficient multimodal embedding pipelines that balance detail retention and storage costs for large-scale image and text data.

Authors

Zijing Cai, Yuzhe Wang, Jingxian Zhu, Fengbin Zhu, Richang Hong

Abstract

Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods either compress each input into a single vector, limiting fine-grained expressiveness, or retain long sequences of visual-token vectors, incurring substantial storage and interaction costs. To resolve this trade-off, we propose ResComEmb, a trainable framework for effective and efficient universal multi-vector multimodal embedding. ResComEmb first encodes each input at native dynamic resolution into ordered global, intermediate, and fine-grained views. After MLLM contextualization and embedding projection, a trainable Residual Homogeneity Compression (RHC) module reduces within-granularity redundancy and cross-granularity repetition under explicit visual token budgets. Then, ResComEmb introduces a length-adaptive Bidirectional Late-Interaction Matching mechanism for robust query-document scoring, which averages the strongest token-level matches in each direction and combines the two scores using a weight based on how many valid tokens each side has. Extensive experiments on MMEB, ViDoRe V1, and ViDoRe V2 show that ResComEmb produces higher-quality universal multimodal embeddings than VLM2Vec-V2, and outperforms ColQwen2.5 in visual document retrieval using only 37.5% of its full visual token budget, demonstrating a favorable effectiveness-efficiency trade-off.