Auxiliary tasks help neural networks learn more effectively
An Analytical Theory of Auxiliary Learning
Machine Learning
Summary
Sometimes neural networks learn better when they try to solve extra tasks alongside their main one. The authors studied why this happens by analyzing how networks learn step-by-step in simple and slightly more complex settings. They found formulas that explain how the relatedness of tasks and noise in the labels affect this improvement. Their work helps show how extra tasks balance the learning and noise to improve final performance.
What this means in practice
- •For machine learning engineers: Improve multi-task training strategies by understanding how auxiliary tasks influence learning dynamics and error reduction.
- •For data scientists: Choose auxiliary tasks that are better aligned with main tasks based on the quantified impact of task correlations and noise.
A theory result. No direct application yet.
Authors
Federico Milanesio, Alessandro Ingrosso, Matteo Osella
Abstract
Auxiliary learning is an optimization paradigm in which a neural network's performance on a target task is improved by jointly training it on additional tasks. However, the mechanisms behind this improvement remain poorly understood. We study this problem using a teacher-student framework and derive a closed system of differential equations describing the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, we obtain a closed-form expression for the generalization error to leading order in the learning rate, quantifying how task correlations and label noise determine the benefit of auxiliary learning. For non-linear activation functions, we develop a fluctuation-dissipation analytical theory that establishes a general relation linking the main and auxiliary errors to the corresponding single-task error. Numerical experiments support the theoretical predictions and show how auxiliary tasks improve generalization by balancing the forcing dynamics towards the optimal solution with gradient noise.