Capacity and Redundancy Trade-offs in Multi-Task Learning
Machine LearningArtificial IntelligenceInformation Theory
Summary
The authors study a problem in multi-task learning where tasks can sometimes interfere and hurt each other's performance, known as negative transfer. They explain this interference using a Capacity–Redundancy identity, which breaks down how much information tasks share and how much conflict remains after sharing. They also find conditions when grouping similar tasks improves learning and justify using gradient similarity as a measure of task redundancy. Their experiments show that a method called clustered LoRA reduces interference and improves results compared to random task grouping.
Authors
Asif Khan
Abstract
In multi-task learning (MTL) negative transfer is often considered as an optimization artifact, but it can also be viewed as a consequence of limited shared capacity and weak task redundancy. We investigate this effect through a Capacity--Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information that includes label redundancy defined via total correlation (TC), and a residual coupling term that quantifies interference left unresolved by the shared representation. Additionally, we show two key results: (i) a clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing, and (ii) a gradient--TC bridge in a Gaussian multi-task model that formally justifies gradient cosine similarity as a proxy for redundancy ordering. Empirically, we estimate the residual coupling $Δ$ from validation residual correlations, showing that clustered LoRA substantially reduces $\widehatΔ$, outperforms size-matched random partitions, and results in statistically significant gains with multi-seed confidence intervals.