A Geometric Perspective on Stabilizing Value Conflict Resolution
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how large language models (LLMs) handle tricky situations where different values conflict, which is hard when models learn from simple human feedback scores. They found that using a step-by-step reasoning method called chain-of-thought (CoT) helps make the model's training process smoother and more stable. By focusing CoT on value conflicts, the authors improved the model's ability to handle moral reasoning tasks. This suggests that designing how the model thinks through problems can help it deal better with complex value conflicts in user requests.
Large Language ModelsReinforcement Learning from Human FeedbackChain-of-Thought ReasoningMoral ReasoningValue ConflictsLoss LandscapeOptimization StabilityPluralistic AlignmentScalar Rewards
Authors
Saket Reddy, Andy Liu
Abstract
Large Language Models (LLMs) often struggle to navigate value conflicts when trained with the compressed scalar rewards of Reinforcement Learning from Human Feedback (RLHF). To address this challenge, we investigate how chain-of-thought (CoT) reasoning can help improve performance in this domain. Geometrically, we show that CoT correlates with further smoothing the model's loss landscape in its sharpest direction, helping resolve the optimization instability of traditional scalar rewards. We also demonstrate via relevant downstream benchmarks that value conflict-focused CoT may generalize to different kinds of moral reasoning, demonstrating that this CoT has the potential to be an effective mechanism for better moral reasoning. To capitalize on this potential, we create a new value conflict-focused CoT design that further smooths the sharpest direction of the loss landscape and increases moral reasoning performance. This finding shows that explicitly modifying and improving the design of reasoning dynamics offers a promising avenue for improving model performance on user requests with complex value conflicts, advancing pluralistic alignment in LLMs.