Ai systems need active questions to find true user preferences
From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction
Human-Computer Interaction
Summary
User actions often help AI guess what people want, but sometimes people’s choices are skewed by strong beliefs, causing the AI to get confused. The authors explain that just watching what users do isn’t enough for AI to separate true preferences from biased actions. They prove that by actively asking users specific questions or creating situations where choices lead to no rewards, AI can better identify real user preferences and avoid harmful feedback loops.
What this means in practice
- •For product designers: Improve AI tools by adding active user queries to detect real preferences beyond biased behavior.
- •For online recommendation teams: Design recommendation systems that intervene with users to reveal true tastes instead of relying solely on observed clicks.
A theory result. No direct application yet.
Authors
Lucas Biechy, Yuxiao Li, Zhonghao He, Tianyi Qiu
Abstract
Alignment methods typically treat user behavior as a reliable proxy for underlying preferences. However, this assumption breaks down when users hold belief-dependent cognitive biases, risking irreversible ``confirmation traps'' and harmful feedback loops. In this paper, we formalize when an AI system can distinguish true user preferences from behavior shaped by cognitive bias. Modeling this interaction as a multi-armed bandit, we first prove a non-identifiability theorem: under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment, forcing any action-only estimator to perform no better than random guessing. We then show that active preference elicitation breaks this observational equivalence. Specifically, we prove that an Observer applying a finite sequence of adversarial zero-reward interventions of length $m_δ$ makes the post-intervention action distributions arbitrarily separable. Our theoretical results suggest that robust alignment requires actively intervening to tell informed preferences from biases.