Multimodal semantic encoding improves image reconstruction over noisy channels
Multimodal Semantic Communication using Shared and Unique Representation Decomposition
Information Theory
Summary
Communicating meaningfully with less data is hard when using different types of information, like pictures and words, together. The paper proposes a method that breaks down information from each type into shared parts and unique parts, so important details aren’t lost. The shared parts use a small, learned codebook to send data efficiently, while unique parts are sent in a different way. Testing on images and text shows better image rebuilding after data noise, compared to older methods.
What this means in practice
- •For mobile network engineers: Enhance image and text data transmission over bandwidth-limited wireless networks with improved reconstruction after noise events.
- •For autonomous vehicle system designers: Improve reliability of multimodal sensory data sharing in vehicular networks to support accurate environment perception under channel noise.
Authors
Peng Yi, Ying-Chang Liang
Abstract
Semantic communication (SemCom) has emerged as a promising paradigm for bandwidth-constrained mobile and vehicular networks. This paper proposes a unified representation learning framework for multimodal SemCom (MSC), which encodes heterogeneous data into a compact semantic space for efficient transmission and source recovery. The framework addresses the multimodal semantic representation gap, which arises when at least one modality contains semantic information that cannot be fully inferred from the other. To preserve this information for reconstruction, the framework decomposes each representation into shared and modality-specific components. We formulate a mutual information objective for source reconstruction, cross-modal alignment of the shared components, and decoupling of the shared and modality-specific components. To enhance transmission efficiency, we introduce a shared knowledge base implemented as a compact codebook of semantic vectors learned during training. The shared components are transmitted digitally as codebook indices, while the modality-specific components are sent in analog form. We apply knowledge distillation during training, using a pretrained vision foundation model to guide the learning of quantized shared features. On image--text pairs over additive white Gaussian noise (AWGN) and Extended Vehicular A (EVA) channels, the proposed framework improves image reconstruction over conventional separate source-channel coding (SSCC) and representative multimodal and unimodal SemCom systems. Further evaluations show gradual degradation under channel-estimation mismatch, complementary effects of alignment, decoupling, and knowledge distillation on the shared representation, and alignment of shared features across modalities with separation of the modality-specific features.