Syn omni improves cross modal embeddings with structured specialization
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
Computer Vision and Pattern RecognitionArtificial IntelligenceInformation Retrieval
Summary
Combining data from images, videos, and sounds into one smart system is tricky because each type of data is different, but they also share some similarities. The authors designed Syn-Omni, a new way to handle this by keeping specific details for each data type separate while still letting them work together. They developed techniques to first focus on individual data types and then gradually mix them for better understanding. This approach was tested on many tasks involving pictures, videos, and audio, consistently doing better than previous methods.
What this means in practice
- •For multimedia software developers: Develop multimedia apps that better integrate images, video, and audio by improving each data type’s specialized processing before combined analysis.
- •For speech recognition teams: Enhance audio-based systems by leveraging structured specialization that maintains audio-specific features while collaborating with other data types for richer context.
Authors
Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji
Abstract
Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.