Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation
2026-07-03 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to best train a single adapter model on multiple domains at once, which is a kind of multi-task learning. Instead of treating all domains equally, they create a method called CoDA that uses simple clues from the model’s current performance on each domain to decide how much focus each domain should get during training. CoDA helps the model learn better by boosting helpful domains and reducing interference from conflicting ones, without extra parameters or labels. Their experiments show CoDA works better than several existing methods across different tasks and models.
multi-task learninglow-rank adapterfine-tuningdomain adaptationcross-domain transferentropy-regularized optimizationgradient conflictLoRAonline data selection
Authors
Wei Zhang, Lin Tang, Ming Zhao, Yuxuan Wang
Abstract
Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain's participation -- jointly its loss weight and its share of the sampled data -- rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.