AI summaryⓘ
The authors studied how large language models (LLMs) hold information about user preferences even before being specifically trained to recognize them. They found that chosen and rejected responses create distinct patterns inside the model, and these patterns get clearer when the model is trained on common preference datasets but disappear when preferences differ from those datasets. Using this insight, they trained a small model on a few labeled examples to identify preferences in a large amount of unlabeled data, helping to improve preference optimization with much less labeled data. Their method worked well across different tasks and models, often better than directly training on the same small data and competitive with much larger labeled datasets.
large language modelspreference optimizationlinear probeintermediate representationsactivation clusteringalignmentlabel efficiencylow-resource learninguser preferencesannotation budget
Abstract
Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation impractical in low-resource settings where preferences cannot be reliably labelled by LLMs themselves, e.g., due to cultural, subjective, or personalised contexts. In this paper, we investigate how language models encode preference information in their intermediate representations, finding that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models. Strikingly, this structure is strengthened by alignment on canonical datasets but erased when the target preferences differ from those the model was aligned on, suggesting aligned LLMs are poor judges for non-mainstream populations. Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs ($\leq$500) and using it to annotate large unlabelled datasets (50K+) for downstream preference optimisation. We systematically evaluate this approach across different datasets, preference optimisation methods and model scales and find that our method consistently outperforms direct training given the same annotation budget, and remains competitive against baselines trained on $50-100\times$ more labelled data in the majority of our settings. Code is available at https://github.com/alessioGalatolo/activ-pref-probe.