Foundation models improve semantic communication at very low bit rates
Foundation Models for Generalizable Semantic and Goal-Oriented Communication
Machine LearningArtificial IntelligenceRobotics
Summary
Sending pictures clearly over very limited data connections is hard because most methods try to send the entire image. The authors created a way that sends only important parts of the image that matter to the goal, then uses smart AI to fill in the rest. This approach avoids learning just from the training images and works well even on new, unseen pictures. Their method keeps the meaning of images with fewer bits and looks visually similar to the originals.
What this means in practice
- •For wireless network engineers: Send high-meaning visual data efficiently over extremely limited 6G channels using sparse semantic anchors and AI-based image completion.
- •For security system integrators: Improve transmission of critical camera images with high fidelity at low data rates to enable reliable remote monitoring under bandwidth constraints.
Authors
Boliang Liu, Wint Yi Poe, Riccardo Trivisonno, Giuseppe Caire
Abstract
Semantic and goal-oriented communication is increasingly studied for 6G, but generalization beyond seen data remains a key weakness under tight rate budgets. Many existing systems overfit their training data and degrade sharply at very low bit rates because they attempt to compress the entire signal. We introduce Foundation Model-Guided Semantic and Goal-Oriented Communication (FMSGOC), a framework that uses broad visual-linguistic Foundation Model priors to mitigate overfitting. It further improves rate efficiency by concentrating bits on sparse, goal-aligned anchors and relying on generative foundation-model priors to reconstruct the masked regions. By decoupling what to send from how to reconstruct, a vision-language foundation model selects and transmits a sparse set of semantic anchors, while a pretrained diffusion model, fine-tuned for masked completion, reconstructs the image at the receiver. In our experiments, FMSGOC reaches 0.039 bits per pixel (BPP), maintains high semantic fidelity (cosine similarity 0.87-0.90 on CIFAR-10), remains robust on previously unseen inputs (0.83-0.86 on ImageNet), and shows good perceptual similarity (0.1278/0.1558, CIFAR-10/ImageNet), outperforming strong end-to-end baselines at lower bit rates.