MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
2026-05-25 • Computation and Language
Computation and Language
AI summaryⓘ
The authors address the challenge of customizing large language models (LLMs) to match multiple user preferences at once without retraining the model. They propose MATO, a method that adjusts how important each preference is during the response generation, based on the user’s goals and partial outputs, without changing the model itself. MATO finds reward signals directly from the LLM and optimizes the balance of different objectives on-the-fly, leading to better alignment with user preferences. Their experiments show MATO improves control and performance compared to existing approaches.
Large Language ModelsMulti-objective OptimizationPersonalized AIPrompt-based PersonalizationTest-time OptimizationReward ModelsControllable GenerationPareto Improvement
Authors
Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen, Junae Kim, Gholamreza Haffari, Dinh Phung
Abstract
Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.