Summary
Multi-task dense prediction involves teaching a computer to recognize and understand multiple visual details all at once, like identifying objects while also noticing their shapes and boundaries. The authors found that existing large vision models struggle when trained to do many things together because their main features conflict with detailed tasks. They created a new method called D2MoE that separates general knowledge from task-specific tweaks, making the model better at handling many tasks simultaneously without losing important details. Their approach keeps the main model frozen and adds small, specialized parts to improve results with fewer changes needed.
What this means in practice
- •For computer vision engineers: Build multi-task models that handle object recognition and structural details together with improved stability and smaller model changes.
- •For autonomous vehicle developers: Enhance perception systems to better interpret complex scenes by combining semantic and geometric understanding from foundation models.
Authors
Yunzhi Zhuge, Xinzhuo Yu, Sitong Gong, Lu Zhang, Pingping Zhang, Huchuan Lu
Abstract
Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.