Self-supervised features transfer better when task preferences align with data structure
Sharp Rates and a One-Line Correction for Spectral Representation Learning
Machine LearningInformation Theory
Summary
When a computer is trained to understand data without specific tasks in mind, it's important to know when its knowledge can be reused effectively for new tasks. The paper shows that the success of reusing learned features depends on how the new tasks' preferences align with the structure of the data the model learned from, not just on assuming all tasks are equally important. The authors provide mathematical rates for how well these features transfer and offer a simple fix to improve them when misalignment is detected. This helps practitioners know when their off-the-shelf models will work well or need adjustment.
What this means in practice
- •For machine learning engineers: Diagnose when pretrained self-supervised models need adjustment before reusing features on new tasks, saving development effort.
- •For computer vision developers: Improve image feature reuse by applying a simple correction to better align pretrained encoders with target task preferences.
Authors
Dier Tang, Jing Yee Tan, Guangyue Han
Abstract
A self-supervised encoder is trained once, frozen, and reused through lightweight probes on tasks nobody named at training time; the practitioner's question is when the off-the-shelf features are good enough and when they need fixing. Canonical correlation analysis, HGR maximal correlation, and the population optimum of the spectral contrastive loss all return the top-$k$ singular subspace of a cross-view dependence operator, justified by isotropy: if the task prior has no directional preference, that subspace is universally optimal. We show isotropy is the wrong hypothesis. The prior enters the transfer risk only through the task covariance $Λ=\mathbb{E}[ΔΔ^\top]$, and only through its compression onto the operator's leading singular directions; what matters is not whether $Λ$ is isotropic but whether its preferred directions are ordered consistently with the operator's spectrum. We prove matching two-sided rates---worst-case regret is exactly $1-1/κ(Λ)$, refines to $1-A_k$ for an alignment coefficient $A_k$, localizes to the top-$2k$ subspace, becomes second order under a spectral gap, and is improvable by no task-agnostic representation---and show why alignment is generic: incoherent preferences cancel in high dimension, and $T$ diverse tasks force $α=\widetilde O(\sqrt{d_x/T})$, a quantitative account of why task diversity, not symmetry, makes self-supervised features transfer. The governing statistics cost $O(kd_x^2)$, and when they signal misalignment a one-line reweighting of the positive-pair term provably restores exact optimality. The result is a diagnostic that answers the practitioner's question from a small labelled budget and refuses when the task bank cannot support the width requested; on controlled data it takes a regret of $0.86$ down to $0.003$, and on a CIFAR-100 encoder it correctly predicts that no correction is needed.