Edustories dataset enables AI evaluation on classroom interventions
Edustories: A Collection of Real-world Case Studies from Classroom Practices
Computation and LanguageArtificial Intelligence
Summary
Classrooms often have many students, but most AI tools focus on helping one student at a time. The authors created Edustories, a big collection of real stories written by teachers about dealing with hard classroom situations and how their actions worked out. They tested modern AI language models to see if they could predict how well these teacher actions would go, comparing AI predictions to expert judgments. The AI did okay but not as well as human experts, showing there is still work before AI can be a helpful assistant for teachers managing whole classrooms.
What this means in practice
- •For educational software developers: Develop AI tools that predict the success of teacher strategies in managing classroom behaviors using the Edustories dataset.$Commercial implications: Enables building AI-driven educational assistant software that offers feedback to teachers on intervention effectiveness, improving classroom management tools.
- •For instructional designers: Create training programs that incorporate real-world teacher case studies from Edustories to improve teacher responses to challenging classroom behaviors.
Authors
Michal Štefánik, Jan Nehyba, Jirina Karasova, Martin Fico, Lucie Škarková, Markéta Košatková, David Kosatka
Abstract
Despite the widely recognized potential of AI in education, most prior work has focused on individualized student assistance. In contrast, the majority of educational practice worldwide still takes place in collective classroom settings. To enable researchers to study AI assistance in collective teaching, we introduce Edustories, a dataset of 1,492 teacher-written case studies describing real elementary and high-school classroom situations involving challenging student behavior, pedagogical interventions, and their outcomes. Among many other applications, Edustories enables evaluating LLMs' ability to predict the success of teacher interventions, crucial for providing practicing teachers with useful feedback. Comparing the latest models from four language-model families against expert assessments, we find that current models fall short of human expertise in predicting classroom outcomes; the strongest models reach 58% accuracy compared to 64% of human experts. This gap highlights both the limitations and the emerging potential of AI as assistants for practicing teachers.