Loss-based training can fail to enforce fair AI reward principles
Enforcing Axioms for AI Alignment under Loss-Based Rules
Computer Science and Game TheoryArtificial IntelligenceMachine Learning
Summary
Many AI systems learn what to do by comparing pairs of responses and deciding which is better. This paper looks at how to make sure these systems always pick responses that are better according to common principles, like being helpful and not toxic. The authors find that even if AI uses more complex models, it can still make choices that don't match these principles perfectly. However, by carefully designing the comparison data the AI learns from, it's possible to make sure it follows all these principles correctly in ideal cases.
What this means in practice
- •For ai alignment engineers: Design training datasets that help loss-based reward models respect multiple ethical principles simultaneously.
- •For machine learning model builders: Understand limitations of certain neural network architectures in achieving principle-consistent reward signals.
A theory result. No direct application yet.
Authors
Alexandros Hollender, Sonja Kraiczy
Abstract
Many alignment methods, most notably reinforcement learning from human feedback (RLHF), train reward models by minimizing a loss on pairwise comparisons of model responses. Inspired by principle-guided methods such as Constitutional AI, we treat each principle as a voter that compares responses according to that principle, for example, preferring the more helpful or less toxic response. We represent these voters as linear directions in a fixed embedding space, a simplifying assumption motivated by the Linear Representation Hypothesis. In this linear social choice model, Ge et al. (2024) showed that an optimal linear reward model can violate Pareto optimality (PO): From the principles-as-voters lens, this means a response A can be less helpful and more toxic than B, yet still receive a higher reward. We ask whether richer reward models or changes to comparison data can recover PO within loss-based training. We show that PO can still fail for polynomials of any fixed degree and one-hidden-layer ReLU multilayer perceptrons (MLPs) of any fixed width $ d\ge 2 $ with suitable finite parameter bounds. We then recover PO for loss-based training of linear rewards under suitable assumptions in an idealized setting with uniform comparison directions and equal comparison lengths. These results motivate dataset design as an approach to recovering PO in constitutional-style alignment while retaining loss-based training.