Large language models spot common student errors in math modelling
Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs
Computers and SocietyArtificial IntelligenceLogic in Computer Science
Summary
Students learning computer science and related fields often make mistakes when writing math-based models like logical formulas or equations. The authors created a process where a large language model (LLM) suggests ways to fix these mistakes, groups similar mistakes together, and then shows these clusters in a visual tool. This helps teachers and researchers see common errors students make. Their method works well on big sets of data and applies to different kinds of math formalisms beyond just logic formulas.
What this means in practice
- •For educational software developers: Create tools that automatically detect and group common student mistakes in formal math tasks to improve feedback systems.
- •For online course platform engineers: Integrate AI-based methods to analyze large student datasets and highlight frequent modeling errors to course instructors.
- •For math tutoring technology makers: Develop commercial products that use AI to diagnose and visualize common errors students make in formal math problems, enhancing tutoring services.$Commercial implications: Enables interactive tutoring products offering targeted error detection and feedback in formal math education.
Authors
Lilian Killich, Marko Schmellenkamp, Fabian Vehlken, Thomas Zeume
Abstract
Modelling with mathematical formalisms like logical formulas, mathematical equations, or regular expressions is an important yet challenging task for students of computer science and other STEM disciplines. Identifying common mistakes occurring in this context is an important step towards helping struggling students by providing targeted high-quality feedback, e.g. in interactive learning systems. We present a tool-supported workflow that allows to (1) identify candidates for common mistakes that explain many student mistakes in large educational data sets, (2) cluster candidates according to similarities, and (3) visualize resulting clusters for instructors and CS education researchers. The visualization is designed to help researchers to identify common modelling mistakes. The candidates for common mistakes are represented by bug fixing transformations that translate incorrect formalizations into correct formalizations; they are generated by an LLM and validated algorithmically. We show that this approach works well by reproducing common mistakes in propositional logic modelling that were identified by hand in the literature; showing that, unlike other algorithmic approaches, the LLM-based approach is suitable for very large sets of data; and applying it to multiple other formalisms to showcase it generalizes beyond propositional logic.