AI scores teacher interactions well for emotional support but less for classroom tasks
I code or AI code: A comparative evaluation of AI-rated scores in classroom observations
Computation and LanguageArtificial Intelligence
Summary
Watching and rating how teachers interact with kids helps improve teaching, but it takes a lot of effort from trained people. The authors tested if a smart AI (GPT-5) could give these ratings from written transcripts. The AI matched humans well when judging how teachers supported kids emotionally and gave helpful feedback. However, it struggled more with evaluating classroom organization and teaching help tasks. This means AI could help as a quick check but isn’t ready to fully replace human observers.
What this means in practice
- •For education administrators: Use AI to preliminarily screen early childhood classroom interactions for emotional support quality to identify where human review is most needed.
- •For online learning platform developers: Incorporate AI scoring as a first step to provide teachers early feedback on their communication and feedback strategies before detailed human evaluation.
Tested on one dataset.
Authors
Y. Fong, J. Xiang, T. Y. D. Chan, K. Lee, E. Y. H. Lau
Abstract
Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet they remain resource-intensive and dependent on trained observers. This study evaluated the feasibility of using a LLM (GPT-5 model) to score teacher-child interactions in early childhood classrooms, benchmarked against human raters. The study analyzed 87 video-recorded observations from 38 classrooms across 30 kindergartens in Hong Kong. Using observation transcripts, the AI model was configured to apply the full Classroom Assessment Scoring System (CLASS) framework. AI-rated scores were then compared with human ratings by examining correlations and differences in mean scores of the CLASS domains and dimensions. The results showed greater convergence between AI and raters for the Emotional Support domain and, in particular, the Quality of Feedback dimension, which captures how teachers use feedback to extend children's learning. Greater divergence emerged for interactions that were more procedural or context-dependent, particularly within the Classroom Organization and Instructional Support domains. These findings suggest that transcript-based AI scoring may capture some of the relative variation in teacher-child interactions but cannot yet reproduce calibrated human judgements consistently across the full CLASS framework. AI-assisted observation may therefore be more appropriate as a preliminary screening tool rather than as a replacement for trained observers, providing teachers with evidence for reflection rather than high-stakes evaluation. Future research should examine whether domain-specific training and incorporation of contextual and visual information can improve alignment between AI and human rated scores.