Large language models vary widely in checking engineering problem solutions
TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics
Computers and SocietyArtificial IntelligenceMachine Learning
Summary
Some computer programs that help students with engineering problems can solve the problems accurately. But solving them isn’t enough. These programs also need to recognize when a student’s work has mistakes and must defend correct answers even if a student disagrees. The authors created a set of 200 engineering problems with student answers to test how well these programs do on solving, spotting errors, and not giving in to false opinions. They found big differences: most programs catch mistakes well, but some have trouble standing up for correct solutions when questioned.
What this means in practice
- •For engineering software developers: Design and test tutoring features in engineering learning tools to ensure reliable answer checking and error detection using this benchmark.
- •For educational technology product teams: Evaluate and select language models for engineering tutoring systems that need to confirm correct student work and detect errors robustly.
Authors
Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad, Emma Lejeune
Abstract
Large language models (LLMs) have rapidly become a default source of help for students in science and engineering courses. In foundational engineering mechanics courses such as statics, students routinely use these tools to check answers, work through partial solutions, and ask for explanations. Effective tutoring in this setting requires more than a correct final answer. A model must also confirm correct work when a student doubts it, and identify errors in flawed work. Existing benchmarks evaluate LLMs primarily on final-answer accuracy, so it is unclear whether models that can solve mechanics problems also have the other technical capabilities that tutoring depends on. Here we introduce TrussTutor-Bench, a benchmark of 200 verified two-dimensional truss problems paired with automatically generated student solutions, and use it to evaluate thirteen LLMs on solution accuracy, sycophancy, and error detection. Baseline accuracy ranged from 64.5% to 97.0%. Every model detected nearly all injected errors in synthetic flawed student work (90.4% to 100%), so error detection did not distinguish models under these clean conditions. Sycophancy did. When a student insisted that correct work was wrong, models rejected correct solutions to problems they had themselves solved in 1.6% to 50.0% of cases. These results show that performance varies widely across models: the leading models solve, check, and defend student work nearly perfectly, while others are far less reliable, particularly when a student doubts correct work. TrussTutor-Bench provides a verified, extensible standard for determining which models are technically reliable enough to put in front of students. Follow-up work must now focus on teaching rather than technical capability, drawing on learning science to determine whether LLM help leads students to understand the material or simply to reach the right answer.