Large language models improve math reasoning by classifying error types
Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
Computation and Language
Summary
Large language models (LLMs) sometimes get math problems wrong because of different kinds of mistakes in their reasoning steps. The authors studied these mistakes by defining and categorizing 21 types of reasoning errors. They found some error types appeared more often and designed a prompt to help LLMs focus on eight common errors. Using this new prompt helped the models perform better at math questions. They also noticed these common errors happened across similarly sized language models.
What this means in practice
- •For software developers: Improve math problem-solving features in AI applications by integrating error-focused prompts that enhance reasoning accuracy.
- •For automated tutoring platforms: Enhance feedback quality by detecting and addressing common reasoning errors in students' math problem solutions generated by AI tutors.
Authors
Runa Yoshida, Kosuke Nishida, Kyosuke Nishida
Abstract
The reasoning ability of large language models (LLMs) is a critical factor for practical LLM-based applications. To investigate the current reasoning capability of LLMs, we clarify the types of errors that arise in LLMs' reasoning processes on mathematical datasets. We focus on problems where LLMs produce an incorrect answer. We define errors in the reasoning process as reasoning errors and manually analyze the features of reasoning errors. We defined and classified 21 error classes and identified the frequently occurring classes among them. Beyond qualitative evaluation, we leverage the evaluation results to improve the reasoning capability. We designed a prompt that explicitly focuses on eight error classes. The experiments demonstrate that this prompt effectively improves reasoning performance. Furthermore, the results suggest that the frequent reasoning errors identified in this paper are common across LLMs of comparable scale.