The Trade-off Between Covariate Dependence and Latent Structure in Representation Learning
2026-08-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study ways to learn hidden features (latent representations) that each match a specific known factor. They explain challenges when these known factors are related, making it hard to get neat one-to-one matches and independent features at the same time. They propose a new supervised method that balances how latent features relate to these known factors and keeps some structure in the data. They also show how to adjust existing models to better align their features with known factors and use a probabilistic model called informed factor analysis to control these features, demonstrating their method on biological data.
Disentangled representation learningLatent representationsCovariatesLatent independenceSupervised learningFactor analysisCLIP modelDINOv2ViT (Vision Transformer)Multi-omics data
Authors
Małgorzata Łazęcka, Ewa Szczurek
Abstract
Disentangled representation learning seeks latent representations whose indicidual dimensions each align with a distinct covariate. Unsupervised approaches typically target latent dimension independence, yet this gives no guarantee that the resulting dimensions align with semantically meaningful covariates. Supervised approaches structure the latent space using observed covariates, but under correlated covariates they cannot simultaneously control one-to-one latent-covariate alignment and latent independence. We introduce a unified, supervised framework that couples latent dimension-covariate dependence with constraints on the latent structure. Within this framework, we show an inherent trade-off, where enforcing latent independence or exclusive one-to-one latent-covariate dependence comes at a provable cost in latent-covariate alignment. We prove that the resulting disentanglement regimes are ordered by the strength of that alignment. Each regime admits a closed-form transformation of the latent space. We apply these transformations post-hoc to realign the representations of pretrained models such as CLIP, DINOv2, and ViT, and we fold them into the inference of informed factor analysis (iFA), a probabilistic model with covariate-informed factors. On simulated and real multi-omics data, we show that both post-hoc alignment and iFA enable controllability of structured latent representations.