Psychometric scale measures how users see value alignment with AI
A Scale For Value Alignment In Human-AI Interaction
Human-Computer Interaction
Summary
Value alignment means how well an AI system’s goals match what people want. The authors found there was no good way to measure how users feel about this match. They created a new questionnaire with 13 questions to check two things: how well the AI understands users’ values and how well it shows those values in action. They tested this questionnaire with over 600 people and confirmed it works well. They also made a shorter version with 6 questions for quick use.
What this means in practice
- •For ai product designers: Use the scale to evaluate and compare how users perceive alignment of AI systems with their values across different products.
- •For ux researchers: Quickly assess user perceptions of value alignment in AI interfaces using the short 6-item questionnaire to guide design improvements.
Authors
Lena Hegemann, Steeven Villa, Hyemin Bang, Mitchell L. Gordon, Antti Oulasvirta, Robin Welsch, Patrick Ebel
Abstract
Value alignment is a central objective in AI and HCI research, yet no validated instrument measures how users perceive it. This gap hampers the comparison and accumulation of findings and limits the effectiveness of applications, where understanding users' viewpoints is critical. We construct and evaluate a 13-item psychometric scale that measures perceived value alignment across two components: value understanding and value manifestation. It is based on a large item pool drawn from prior empirical studies, filtered by experts, and finally assessed by users (N=607) across diverse AI scenarios. Confirmatory factor analysis on an independent sample (N=259) confirmed the two-factor structure and high internal consistency for both subscales. Using optimization, we also derived a 6-item short form for quick administration. The scale is a reliable measure, providing HCI researchers with a common evaluation metric across contexts.