Multimodal model improves test-time adaptation with representation editing
Representation Editing for Multimodal Test-Time Adaptation
Machine Learning
Summary
Multimodal models combine information from different types of data, like images and text, but can struggle when new data looks different from what they were trained on. The authors developed a method called FIRE that adjusts the model’s internal layers on the fly using unlabeled test data, improving how the model aligns information from each data type before combining them. This approach uses a special frequency-based technique to make the adjustments more stable and effective. As a result, the model makes more reliable predictions even when faced with new types of data problems.
What this means in practice
- •For autonomous vehicle teams: Improve sensor data fusion reliability in real time under diverse environmental conditions without requiring labeled data.
- •For surveillance system engineers: Enhance multimodal sensor integration to maintain performance when cameras and microphones experience varying noise or distortions.
Authors
Longfei Huang, Xiangyu Wu, Yang Yang
Abstract
Multimodal test-time adaptation (TTA) aims to adapt a pretrained multimodal model online to distribution shift across modalities using unlabeled test data, showing broad potential in real-world applications. However, existing methods primarily focus on adjusting fused features to bridge the source-target gap, lacking explicit control over intermediate representation misalignment, which is a key driver of performance drop under distribution shift. In this work, we tackle this challenge from the perspective of representation engineering. Unlike previous TTA methods that update fusion weights in place, we propose FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations. Specifically, we first adopt representation editors into each intermediate layer of the unimodal encoders, enabling layer-wise calibration of unimodal representations. To further enhance the diversity and stability of the low-rank editing subspaces, each representation editor performs frequency domain mixing via the fast Fourier transform to construct structured bases. Moreover, we introduce multi-level adaptation objectives to optimize these editors, jointly promoting cross-modal semantic alignment, source-target statistical alignment, and asymmetric prediction consistency. In this way, FIRE yields aligned unimodal representations for fusion and further improves prediction reliability. Extensive experiments on two widely used multimodal benchmarks under various corruption types demonstrate the superiority of FIRE over existing multimodal TTA methods.