Spectral method improves flexible compact data representations
SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum
Computation and Language
Summary
Representing data in flexible sizes helps computers use memory more efficiently. The authors improved an existing method called Matryoshka Representation Learning by organizing data features using a new spectral approach that prioritizes important features. This lets smaller selected parts of the data still give a good summary. Their method, SpecMRL, did better than the original on tests, especially when using very small sizes. It can even beat the original method that uses all data features while using only half the size.
What this means in practice
- •For machine learning engineers: Build models that adaptively compress data features while keeping accuracy by using spectral ordering to select compact embeddings.
- •For mobile app developers: Create apps that use smaller, more efficient data representations to reduce memory and compute while maintaining quality.
Authors
Zhongxin Chen, Songhao Wu, Heng Cui, Cong Li, Xiao Zhou, Rui Yan
Abstract
Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.