Attention methods keep results stable despite signal changes in transformers
Refinement Symmetry in Multimodal Transformers
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Attention in multimodal transformers depends on how many pieces (tokens) a signal is split into, which can change its influence unexpectedly. The authors studied how breaking signals into smaller parts while keeping content and position the same should not change results. They developed a method that weighs attention to keep answers consistent even when the tokens representing images or videos are duplicated or merged differently. Their approach improves stability and accuracy in real models handling video and multimodal data.
What this means in practice
- •For multimodal ai engineers: Maintain consistent model outputs when changing how inputs like videos or images are tokenized or sampled during training or deployment.
- •For video processing teams: Reduce errors and improve stability in video-based AI systems by ensuring attention weights remain stable despite frame resampling or compression.
Authors
Yuhao Du, Shunian Chen
Abstract
Attention weights depend on token counts, which change with the representation of a signal. We study refinement symmetry: splitting a representation while preserving content, position, visible context, and total mass should preserve its contribution. Building on proportional and quadrature attention, we show that split invariance forces the local mass factor to be linear for any fixed positive attention kernel, provided that factor is nondecreasing. For changed representations, a physical coupling bounds attention error by separating feature change from weight reallocation. In Qwen2.5-Omni-7B, duplicating half the visual tokens threefold changes 255 of 3,586 MVBench answers under standard attention; measure weighting preserves every answer under matched visibility. Under natural frame resampling, it reduces distributional drift. At twofold merging of a frozen video encoding, a five-seed evaluation shows an all-partition-correct accuracy gain of 1.04 percentage points over global count weighting (average group mass) and 0.93 points over standard attention. The advantage over global count also holds on WorldSense but depends on the compression budget. The result is a representation principle with a measured benefit in robustness across partitions.