TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment
2026-06-01 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors studied how personalized large language models can sometimes give less accurate answers to certain social groups, causing fairness problems. They propose a new method called TriAlign that treats different social groups as separate agents and uses reinforcement learning to improve accuracy, consistency of facts, and personalization all at once. Their approach reduces the differences in truthfulness between groups while still tailoring responses to users. Experiments showed that TriAlign does better than other methods in balancing these goals.
large language modelspersonalizationuniversal truth consistencyfairnessmulti-agent reinforcement learningalignmentcross-group disparitiesobjective task accuracysocial groups
Authors
Thi-Nhung Nguyen, Linhao Luo, Rollin Omari, Junae Kim, Thuy-Trang Vu, Dinh Phung
Abstract
Personalized large language models adapt responses to users' preferences and social attributes, but can introduce substantial universal truth inconsistencies across social groups, where some groups systematically receive less accurate responses on objective tasks. Existing alignment methods either ignore personalization or mainly focus on subjective preference alignment, largely overlooking fairness and consistency in universal truths. To address this gap, we study Truth-Invariant Alignment (TIA), an alignment problem for personalized LLMs that aims to ensure universal truths remain consistent across social groups while preserving personalization. We propose TriAlign, the first offline multi-agent reinforcement learning (MARL) framework for TIA, where each social group is modeled as an agent interacting. TriAlign jointly optimizes universal truth accuracy, cross-group truth consistency, and personalization through a fairness-aware objective and an explicit inconsistency penalty. Experiments across diverse benchmarks demonstrate that TriAlign achieves a stronger balance among these three objectives than strong baselines, reducing universal truth disparities across social groups while improving both objective task performance and personalization quality.