A Method for Learning Value Systems in Generative AI
Computers and SocietyArtificial IntelligenceComputation and LanguageMachine Learning
Summary
The authors focus on teaching AI systems to understand and prioritize human values better by observing human preferences in their responses. They adapt an existing method to help generative AI not just copy preferences, but learn a clearer picture of different values and how to combine them. Their approach uses comparisons of AI responses to learn both what the values mean and how important each value is. Tests show their method works well and makes the AI’s value judgments easier to understand compared to other methods.
Authors
Andrés Holgado-Sánchez, Holger Billhardt, Sascha Ossowski
Abstract
Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours. As such representations are difficult to elicit, value learning seeks to infer them by observing human behaviour. This work addresses the lack of grounded value learning methods in generative AI: existing approaches typically replicate human preferences without awareness of the multidimensional structure of value alignment, or lack principled value system elicitation methods. To address these gaps, we adapt a previously validated value system learning method to the generative AI setting, which, based on pairwise prompt-response preference data, simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model, and ii) a value system representation in the form of a weighted linear scalarization of the previous grounding model. To ensure that the learned value systems are based on coherent value representations, our algorithm dynamically prioritizes the grounding learning process. We evaluate the method against baselines and a contemporary method on prompt-response preference datasets. Results show competitive performance and minimal trade-offs against the baselines, while improving explainability.