Reasoning formats affect how people judge AI answers and trust them
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Machine LearningHuman-Computer Interaction
Summary
People use different ways to show how AI models get their answers, called reasoning representations. This study by the authors tested six such formats to see how well they help people understand, check for mistakes, and trust AI answers. They found that while people liked complex plans and breakdowns, simpler step-by-step explanations actually made it easier to verify and trust the answers. Interestingly, preferred formats sometimes led people to doubt correct answers or trust without checking. So, what people like doesn’t always match what helps them evaluate AI well.
large language modelsreasoning representationchain-of-thoughtexplainabilityhuman evaluationtrust calibrationerror detectioninterpretability
Authors
Jaewoo Lim, Sungbok Shin, Sanghyun Hong
Abstract
Reasoning representations are increasingly used as explanations for large language model outputs. Yet they are typically evaluated with model-centric criteria, such as answer accuracy and faithfulness, leaving it unclear whether they help people evaluate model responses. In this work, we study reasoning representations as human-facing interfaces rather than proxies for model reasoning ability. We conduct a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order. The study collects fine-grained judgments of structural understanding, error detection and localization, and trust calibration. Our study shows a mismatch between perceived preference and support for human evaluation. Participants prefer planning- and decomposition-based representations, but simpler chain-of-thought traces better support verification, trust, and interpretability. Preferred representations also introduce calibration risks, with more false alarms on correct traces and high trust despite low willingness to verify.