Multi-task learning improved by scale-invariant loss balancing method
SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning
Machine LearningArtificial Intelligence
Summary
When computers learn to do several tasks at once, they often struggle because some tasks have bigger numbers than others, making the training unfair. The authors found that existing ways to combine tasks during learning get tricked by these big differences in numbers. They created a new method that uses math to keep the learning fair no matter the size of these task numbers by turning them into a form that treats them evenly. This method works better in tests and helps computers learn multiple tasks more effectively.
What this means in practice
- •For machine learning engineers: Improve training of multi-task models by balancing task losses regardless of their scales to get better overall performance.
- •For data scientists: Use scale-invariant scalarization to handle diverse task objectives without bias toward larger loss values in multi-task prediction problems.
Authors
Zebin Chen, Fei Xing, Yang Chen, Hua Liu, Andy HF Chow, Yuhua Qian, Yu Zhang
Abstract
Multi-task learning (MTL) requires navigating unavoidable trade-offs among competing objectives. This paradigm is frequently formulated as multi-objective optimization (MOO), where the scalarization is favored to reduce an MOO problem to a single objective. We empirically find that existing merit-function-based scalarization approaches are sensitive to the relative scales of different objectives in practical MTL, where task losses commonly differ by orders of magnitude. The optimization process often favors objectives with larger scales even though the underlying Pareto optimal solutions remains invariant to rescaling (i.e., multiplying an objective by a positive constant). To address this issue, we propose Scale-Invariant Merit-function-based Scalarization (SIMS) for MTL. Specifically, SIMS adopts a transformation-induced merit function to convert the MOO problem of MTL to a single objective that renders optimization invariant to the magnitudes of losses. Theoretically, we prove that the requirement for scale invariance uniquely determines this transformation to be logarithmic. We further show that this general transformation-induced merit function preserves weak Pareto optimality and admits a smooth surrogate with controllable approximation error. Extensive experiments on representative multi-task benchmarks demonstrate that SIMS consistently outperforms existing scalarization methods and achieves state-of-the-art performance.