Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explain that humans are important at every step of creating AI, but people often disagree with each other and even themselves over time. These disagreements come from personal opinions and different backgrounds, especially in areas like AI safety and content moderation. Current methods for checking AI often ignore these differences by focusing only on majority opinions, which hides minority views and causes problems with repeating results in AI studies. The authors propose a new way to evaluate AI that includes different human perspectives to make AI systems more trustworthy and aligned with various human values.
AI alignmenthuman-centered AImodel evaluationhuman disagreementplurality votingreproducibility crisissentiment analysiscontent moderationminority perspectivesmachine learning evaluation
Authors
Deepak Pandita, Christopher M. Homan
Abstract
Humans play a vital role at every stage of AI development, from data collection and curation to model development and evaluation. However, humans often disagree with each other and sometimes with themselves over time. It is essential to take disagreement into account when building human-centered AI systems, especially in domains where it is prevalent, such as AI safety, content moderation, or sentiment analysis. Disagreement often arises from subjective human opinion and can vary with one's identity, beliefs, and social environment. Despite this, current LLM evaluation approaches frequently rely on aggregating labels (often via plurality voting) to represent consensus, thereby obscuring minority perspectives. By failing to account for human disagreement, these evaluation methods contribute to the reproducibility crisis in AI. Human feedback is also crucial for ensuring that AI systems align with human values. For these systems to be trustworthy, it is critical to ensure that they reflect diverse human values and perspectives. In this thesis proposal, we present a human-centered and perspective-aware framework for reproducible ML evaluation and AI alignment.