PAIQ improves image understanding by combining visual models with rotation
PAIQ: Patch-Aligned Semantic Injection via Residual Rotation
Computer Vision and Pattern Recognition
Summary
Understanding images well needs both recognizing small details and big ideas. The paper presents PAIQ, a way to mix two types of artificial vision models so they work better together. It cleverly adjusts combined features using a special rotation method to keep differences clear. This method helps systems describe images and answer questions about them more accurately without confusing details. It works well with different language models and beats other ways of mixing visual information.
What this means in practice
- •For computer vision engineers: Build image description and question answering tools that combine detailed and abstract visual features more effectively using PAIQ's rotation-based fusion.
- •For language model developers: Enhance multimodal interfaces by integrating fixed visual encoders with language backbones through PAIQ to reduce errors in visual understanding.
Authors
Pinze Ren, Yuwei Zhang, Hao Chen, Linghao Meng, Chang Li, Qiankun Li
Abstract
Language-aligned and self-supervised visual encoders offer complementary strengths in semantic abstraction and spatial detail. Harnessing this complementarity requires enriching local features while retaining distinctions between semantically related patches. We introduce PAIQ, a patch-aligned semantic injection framework that combines content-based cross-encoder matching with orthogonally constrained residual updates. Using DINOv3 patch features as the spatial base, PAIQ aggregates complementary SigLIP features through joint source allocation and injects the aggregate--base differences through a shared orthogonal transformation Q. This rotation adapts update directions while preserving residual norms and pairwise angles. For fixed projected features, we derive conditions for patch separability under similar semantic aggregates and show that rotation adds a nonnegative separation term over direct interpolation when the aggregate is shared. Only the projection and fusion parameters are trained; both visual encoders and the language model remain frozen, and fusion retains 196 visual tokens. Across diverse language backbones, PAIQ yields broad gains in judge-assessed correctness and reductions in hallucination severity over single-encoder interfaces on image description and visual question answering. On the 2B and 9B Qwen backbones, this compact interface outperforms the strongest evaluated fusion or token-compression baselines by about 2.9 correctness points on average.