Reasoning formats affect how people check AI model answers
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Machine LearningHuman-Computer Interaction
Summary
People need ways to understand and trust answers from large language models (LLMs). The authors studied six different ways to show the reasoning behind LLM answers to see which best helps people check and trust those answers. They found that while many people liked clear, structured plans, the simplest step-by-step reasoning helped them spot errors and decide whether to trust the answer better. Some preferred formats made people too confident or triggered unnecessary doubts, showing a mismatch between what users like and what works best for verifying AI outputs.
What this means in practice
- •For ai product designers: Design user interfaces that show simple step-by-step reasoning to help users verify AI answers more effectively.$Commercial implications: This paper enables AI product designers to create trustworthy explanation formats that improve user verification and trust in AI outputs.
- •For quality assurance teams: Use different reasoning display formats to better detect and localize errors in AI-generated responses across many tasks.
Authors
Jaewoo Lim, Sungbok Shin, Sanghyun Hong
Abstract
Reasoning representations are increasingly used as explanations for large language model outputs. Yet they are typically evaluated with model-centric criteria, such as answer accuracy and faithfulness, leaving it unclear whether they help people evaluate model responses. In this work, we study reasoning representations as human-facing interfaces rather than proxies for model reasoning ability. We conduct a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order. The study collects fine-grained judgments of structural understanding, error detection and localization, and trust calibration. Our study shows a mismatch between perceived preference and support for human evaluation. Participants prefer planning- and decomposition-based representations, but simpler chain-of-thought traces better support verification, trust, and interpretability. Preferred representations also introduce calibration risks, with more false alarms on correct traces and high trust despite low willingness to verify.