Interpretable coding of educational talks using observable behaviors in language models
EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues
Computation and LanguageComputers and SocietyMachine Learning
Summary
Classifying educational conversations is hard because large language models (LLMs) don’t explain why they pick certain labels for a statement. The authors introduce EduBehaviors, which uses LLMs to find clear, repeated behaviors in dialogues and then builds a classifier based on these behaviors. This makes the labeling process more understandable and still competitive in accuracy. They tested their method on a dataset about teacher talk types and created tools so others can use this method on their own data.
What this means in practice
- •For educational technology developers: Create tools that automatically and transparently classify teacher interactions in real-time to support formative feedback.
- •For dialogue system engineers: Integrate interpretable behavioral features extracted by LLMs to improve conversational agents’ monitoring and feedback capabilities.
Authors
Julian Bernado, Ana Trindade Ribeiro, Xander Beberman, Susanna Loeb
Abstract
Large language models have allowed the rapid deployment of pedagogical annotations corresponding to constructs of interest, allowing a natural language interface for generating classifications on a conversational dataset. However due to the opaque nature of LLM reasoning, we have no verifiable, mechanistic insight into why a model chose a label for an utterance. We introduce the EduBehaviors framework, an interpretable, scalable approach to annotating educational data that uses LLMs to measure repeated observable behaviors relevant to many constructs of interest and then learns a classifier for the construct based on these observable behaviors. We evaluate the framework on the TalkMoves dataset, predicting the Teacher TalkMoves labels. Our best configuration results in a macro-F1 of 0.673 and 0.688 Cohen's kappa, proving competitive with direct prompting approaches. In addition, we release EduBehaviors Toolkit, two tools allowing researchers to operationalize the EduBehaviors framework in their own data.