Vision transformers improve multi-channel image understanding with token decorrelation
Disentangling Channel Semantics in Vision Transformers via Token Decorrelation and Composition-Aware Modulation
Computer Vision and Pattern Recognition
Summary
Multi-channel images contain different types of information that are aligned over the same space, so models need to understand both shared and unique parts correctly. The authors found that existing vision transformers tend to overly focus on some channels and ignore others, leading to poor summaries. They created a new method called MuCa-ViT that helps the model keep tokens independent and adjust channel interpretation dynamically. This approach performs better on several multi-channel image datasets, especially when channels have overlapping information.
What this means in practice
- •For remote sensing teams: Improve multi-channel satellite image analysis by better handling channel-specific and shared information for more accurate interpretation.
- •For medical imaging engineers: Enhance multi-channel medical image classification by explicitly separating global and channel-specific features to reduce redundant representations.
Authors
Daeun Kim, Hyesong Choi, Hyejin Park, Dongbo Min
Abstract
Multi-channel images encode heterogeneous channel semantics aligned over a shared spatial structure, which requires models to jointly capture globally shared structure and channel-specific variations. Recent multi-channel Vision Transformers attempt to address this challenge by augmenting the [CLS] token with memory tokens to organize multi-channel representations under varying channel compositions. However, we find that these context tokens collapse onto a few dominant channels under high inter-channel redundancy, producing redundant representations and a biased global summary that fails to reliably support composition-aware channel interpretation. We propose MuCa-ViT (Multi-Channel Composition-Aware Vision Transformer), a unified framework that addresses these limitations through two coupled mechanisms. Factorial Token Learning (FTL) enforces token-level decorrelation among context tokens, encouraging the [CLS] token to capture globally shared semantics while memory tokens preserve complementary channel-specific cues. Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] as a semantic anchor to generate sample-wise modulation signals, which enable composition-aware interpretation of channels within a unified backbone rather than relying on static channel embeddings. Experiments on JUMP-CP, CHAMMI, and So2Sat show that MuCa-ViT outperforms prior multi-channel ViTs across all three benchmarks, and the magnitude of improvement aligns with the inter-channel redundancy of each dataset ($ρ\in [0.19, 0.77]$), validating the diagnostic analysis that motivates our design. Our code is available at https://github.com/kimdanni/MuCa-ViT .