Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors introduce Self-Routed Tensor Adapters (SRTA), a new method to adapt large visual models for multiple different image domains efficiently. Unlike previous approaches that use fixed adaptation or separate routing systems, SRTA creates input-specific adaptations by blending shared components based on the input itself. This allows the model to specialize for different styles or contexts while reusing common features, using fewer trainable parameters than other expert-based methods. The authors show that SRTA performs well on several multi-domain image classification tasks with less complexity.
visual foundation modelsparameter-efficient fine-tuninglow-rank adaptersmixture of experts (MoE)Tucker decompositionrouting mechanismsmulti-domain adaptationmulti-domain classificationtrainable parameterssample-specific adaptation
Authors
Suraj Yadav
Abstract
Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules. Parameter-efficient fine-tuning adapts frozen visual foundation models efficiently, but standard low-rank adapters use a fixed subspace for all inputs, which can be restrictive when domains differ in style, background, and semantic context. MoE-based adapters improve specialization through multiple expert pathways, but often rely on external routers and large expert banks, adding parameters and separating routing from adaptation. We propose \textbf{Self-Routed Tensor Adapters}, a compact framework for multi-domain visual adaptation. SRTA projects each input into a low-rank space, computes routing weights from this representation using a learnable domain matrix, and uses these weights to blend slices of a shared Tucker core. This produces a sample-specific adaptation matrix without an external gating network, allowing shared visual factors to be reused while supporting domain-aware specialization. To strengthen pathway learning, we introduce a progressive depth-weighted routing objective that supervises routing decisions across adapter layers. Across five heterogeneous multi-domain visual classification benchmarks, SRTA achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines while using substantially fewer trainable parameters. At rank 64, SRTA uses 2.77M parameters in the 4-domain setting compared with 9.52M for MoLoRA, and 3.00M in the 6-domain setting compared with 14.31M. Overall, SRTA offers an effective accuracy-parameter trade-off for adapting visual foundation models toward universal multi-domain representations. \href{https://github.com/surajyadav-research/SRTA}{GitHub}