Pre-trained models show varied success assessing AI-made educational questions
Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
It’s getting easier for AI to create educational questions, but checking how good those questions are is harder. The authors studied different AI models that classify questions by their difficulty or thinking level to see which work best on new, AI-generated questions. They found that simple machine learning methods struggle, while large language models do better. Adding extra information or retraining models helps improve accuracy. This work shows some ways to better judge AI-made educational materials automatically.
What this means in practice
- •For educational technology developers: Improve automatic evaluation tools for AI-generated learning questions by choosing better models and data strategies to maintain accuracy across new question types.
- •For nlp platform engineers: Enhance natural language understanding components by using text splicing and learning objective inputs to stabilize performance on shifted datasets in educational tasks.
Authors
Michael Lawrence Castanares, Princess Ventures, Allan Tan
Abstract
The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using Bloom Classifier models is a promising approach to assess educational materials at scale. These models show high accuracy within-distribution dataset (IID Dataset). However, applying the same models to new out-of-distribution (OOD) datasets such as AI-assisted generated questions could show performance degradation. To identify robust classifiers under dataset shift, we evaluated traditional Machine Learning (ML), transformer, and Large Language models on the Bloom level classification task. We also explored feature-engineering strategies incorporating NLP metrics, appending the learning objectives as part of the input, and text splicing to stabilize OOD performance. Our baseline tests show that TFPOS-IDF ML models perform poorly on OOD (Macro F1-score 0.48) compared to BERT (0.55) and LLMs (0.79). Text splicing improved macro F1-score performance of ML and BERT models (0.59 and 0.62, respectively). Appending the learning objectives with the input increased model performance on specific dataset. Model retraining provided the largest improvement across models and datasets. Overall, these findings highlight the trade-off on the use of pre-trained models with novel AI-assisted educational questions and how strategic feature enhancements help address loss in performance.