Vision transformer backbone improves multi-task recognition accuracy
SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition
Computer Vision and Pattern Recognition
Summary
The paper presents SAGE, a method to make large vision models better at recognizing different visual tasks like detecting objects, segmenting images, and identifying key points. The authors introduce techniques that help the model focus on important image parts and details during training. Their method leads to better accuracy on multiple datasets using the same main model. This approach also works efficiently on specialized AI hardware.
What this means in practice
- •For computer vision engineers: Use a single vision transformer backbone for high-accuracy object detection, segmentation, and keypoint estimation across multiple datasets.
- •For ai hardware developers: Deploy multi-task vision recognition models optimized for Huawei Ascend NPUs with state-of-the-art accuracy.
Authors
Le Yang, Yucheng Huang, Junyu Chen, Qi Dang, Xin Kong, Haifeng Huo, Qiang Zhao
Abstract
When adapting self-supervised Vision Transformers (ViTs) to dense prediction, existing methods often treat the ViT as a passive feature extractor and may underuse the rich priors encoded in its representations, including saliency, global context, and high-frequency details. We propose SAGE (Saliency-Annealed Gated Encoder), an enhanced ViT-Adapter backbone that explicitly exploits these priors through three complementary innovations: (1) a saliency-guided architecture that extracts DINO-style class-token (CLS) saliency to guide feature gating and high-frequency enhancement, together with dynamic multi-layer aggregation and a local-detail branch; (2) an annealing-based training paradigm designed to improve convergence and feature quality during early training while reducing feature drift later in training; and (3) a two-stage, quality-aware training mechanism that smoothly transitions from focal loss to Varifocal Loss (VFL) within a single training run. Across the evaluated tasks (Fig.~\ref{fig:fig_com}), SAGE achieves state-of-the-art (SOTA) or competitive results on COCO (62.8 box AP and 55.3 mask AP), COCO keypoint estimation (78.5 AP), ADE20K (60.3 mIoU for semantic segmentation and 44.9 mask AP for instance segmentation), TinyImageNet classification (94.76\% Top-1 accuracy), and DIOR-R oriented object detection (78.8 AP$_{50}$). The same backbone is used across tasks, with task-specific decoders. To the best of our knowledge, SAGE is the first framework to achieve multi-task state-of-the-art results on Huawei Ascend NPUs. Code and pretrained models will be released after internal review.