Multi-task learning improved through shared low-rank parameter updates
CRLoMTL: Conflict-Resolution through Low-rank matrices for Multi-Task Learning
Computer Vision and Pattern Recognition
Summary
Multi-task learning trains one model to do several tasks at once, but sometimes the tasks interfere and make the model worse. The authors propose a new way called CRLoMTL that helps the model avoid conflicts by learning small task-specific updates in a shared low-dimensional space. This approach combines the updates efficiently without changing the model’s usual structure when running it. Tests on vision tasks show that CRLoMTL improves accuracy while training fewer parameters than doing each task separately.
What this means in practice
- •For computer vision engineers: Train multi-task vision models with fewer parameters and improved accuracy by using shared low-rank update techniques.
- •For machine learning platform teams: Deploy efficient multi-task models that maintain original inference speed by integrating low-rank fused updates without added runtime complexity.
Authors
Ahmed Agiza, Tasneem Shaffee, Rahil Badkul, Sherief Reda
Abstract
Multi-task learning (MTL) improves model efficiency by sharing representations across related vision tasks, but it often suffers from negative transfer when task updates interfere with one another. Existing conflict-mitigation methods typically operate in gradient space, requiring explicit task-gradient manipulation and additional optimization steps. We propose Conflict-Resolution through Low-Rank Matrices for Multi-Task Learning (CRLoMTL), a parameter-efficient framework that mitigates task interference in the low-rank parameter-update space. CRLoMTL learns task-specific LoRA updates through task-specific passes, where each task updates its own low-rank matrices, and integrates these updates through global shared-subspace fusion. Specifically, we introduce a shared-subspace resolver based on singular value decomposition (SVD), referred to as the SVD resolver, which extracts a common low-dimensional orthonormal basis across task adapters and projects task updates into this shared space before merging them into a single deployment-time update. The resulting update can be folded into the backbone, preserving the inference structure of the original model without additional runtime branches. Experiments with Swin Transformer backbones on PASCAL-Context, NYUD-v2, and CelebA show that CRLoMTL achieves a favorable accuracy-parameter trade-off, improving over single-task baselines by up to 2.77% on PASCAL-Context, 5.03% on NYUD-v2, and 3.50% on CelebA, while training only a small fraction of the parameters used by single-task models.