AI prefers logic while users engage more with emotion in answers
Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement
Computation and Language
Summary
People and AI disagree on what makes answers interesting online. The paper finds that AI models tend to add a lot of logical details to make answers look better, but real users prefer answers that show more feeling and personality. To fix this, the researchers created a method that reduces over-explaining by the AI while keeping facts and clear opinions. Their method made AI answers closer to what real users like and even won more human preference votes in tests.
What this means in practice
- •For content platform operators: Improve AI-generated answer quality to better match real user engagement by balancing logic and emotional expression.
- •For chatbot developers: Use the OMRA method to reduce AI over-explaining in generated responses, making interactions feel more natural and emotionally engaging.
Authors
Xinglang Zhang, Yuanmeng Xiang, Yunyao Zhang, Zeliang Chen, Junqing Yu, Zikai Song
Abstract
Large language models are increasingly used to generate and evaluate online content, yet it remains unclear whether the qualities they associate with higher engagement match what real users respond to. We study this question using 1.17 million answers to 25,978 questions from Zhihu, Quora, and Reddit, comparing real platform answers and AI-generated answers across four within-question engagement levels. We introduce Ontological Preference Measurement, which represents answers along three dimensions: logic, affect, and expression. We find a systematic gap between AI preference and real user engagement: as target engagement increases, LLMs add more explicit logical structure, while real user engagement is more strongly associated with affective and expressive salience. We call this tendency logic overbinding. Based on this diagnosis, we propose Ontology-Masked Reasoning Autoencoding (OMRA), a controlled intervention that masks and reconstructs over-explained spans while preserving stance, factual content, and coherence. Across four LLM families, OMRA reduces the measured gap by an average of 54.4%. In human evaluation, OMRA wins 62.4% of pairwise preference judgments against matched real platform answers, even though the real answers are more often judged to be human-written.