LLM personalization improves by learning user preferences beyond imitation
MIRROR: From Imitation to Internalization in LLM Personalization
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) are being personalized to better match individual users, but just copying writing style isn’t enough for good content quality. The authors studied a new way called MIRROR that helps models internalize what users prefer instead of just imitating example texts. They also created MIRROR-F, which focuses on important words to keep the content good while still sounding like the user. Their methods worked better than older ones on several tests, producing higher-quality personalized text without losing useful general knowledge.
What this means in practice
- •For software developers: Build smarter personalized chatbots that generate content reflecting user preferences more effectively.
- •For content platform engineers: Enhance recommendation engines to produce user-tailored textual content without compromising quality or original model knowledge.
Authors
Huayi Lai, Jicheng Yang, Min Yi, Chong Meng
Abstract
The demand for personalized LLMs is shifting from style imitation toward content quality. We investigate whether self-distillation can bridge this gap in existing fine-tuning paradigm. To address this limitation, we introduce MIRROR(Meta- personalization by Internalizing Reference-Revealed On-policy Reflections), a novel self-distillation framework that shifts LLM personalization from imitation toward preference internalization. First, we replace reference-token imitation with reference-revealed on-policy self-distillation, aligning the model's next-token distributions along its own generation trajectories with those of its reference-conditioned self, thereby internalizing user preferences rather than reproducing reference wording.Second, we introduce MIRROR-F, a focal plug-in that augments on-policy distributional alignment with selective supervision over informative reference tokens, thereby strengthening content generation while preserving user-specific expression. Across three personalized generation benchmarks, two model scales, and complementary reference-based and LLM-based evaluations, MIRROR and MIRROR-F achieve leading overall personalization performance and superior text quality, while exhibiting less catastrophic forgetting than SFT-based baselines on three unseen personalized generation tasks. The gains are consistent across model scales and application scenarios, translating to improved performance in LLM personalization tasks.