Self-supervised method speeds up text embedding fine-tuning without negatives
Cluster Validation Indices as Self-Supervised Objectives for Text Representation Learning
Computation and LanguageArtificial Intelligence
Summary
Fine-tuning language models to better represent text usually needs extra data tricks or complex setups, which can be slow and use a lot of memory. The authors show a method called SilK that groups similar documents and improves embeddings by focusing on how well those groups form, without needing negative examples or extra views. This method is faster and uses less memory than existing ones, while still performing well on tasks that use these embeddings. It simplifies the process of improving text representations, making it more efficient.
What this means in practice
- •For natural language processing engineers: Improve embedding fine-tuning speed and reduce GPU memory use in language model workflows without needing auxiliary negatives or augmentations.
- •For data labeling teams: Deploy embedding fine-tuning for text data when labels or contrastive pairs are unavailable, enabling better downstream task performance.
Authors
Kishor Kumar Bhaumik, Nicolas Roque dos Santos, Neil Shah, Jia Chen, Evangelos E. Papalexakis
Abstract
Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SilK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SilK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SilK trains 1.46x faster per epoch than the fastest baseline we evaluate and uses 45.4% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SilK stays competitive with the best baselines on three downstream tasks.