Disengagement-aware simulators improve evaluation of AI tutors
Simulating Disengaged Students to Evaluate LLM-based Tutors
Machine Learning
Summary
Sometimes students using AI tutors lose focus or try to trick the system. The authors created computer models that mimic different types of disengaged student behaviors so AI tutors can be tested more realistically. Their models matched human judgments well and helped reveal how various AI tutors perform with different student behaviors. This approach helps developers understand and improve AI tutoring systems before they are used by real learners.
What this means in practice
- •For ai tutoring system developers: Evaluate and improve AI tutoring systems by testing their responses to realistic disengaged student behaviors before deployment.
- •For educational software companies: Benchmark AI tutor products across various learner engagement states to better tailor support strategies in commercial offerings.$Commercial implications: Enables development of marketable AI tutors with validated performance for diverse learner behaviors.
Authors
Xianghui Meng, Jionghao Lin
Abstract
Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.