Multi dimensional educational data scoring improves language model training
Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements
Computation and LanguageArtificial Intelligence
Summary
Many language models learn better when trained on good educational content, but not all learning materials are helpful in the same way. The authors created a tool called Edu-QuRating that checks educational texts on several qualities like accuracy and engagement, rather than just one score. This helps pick better study materials for training small language models, making them perform better on some tests. They also used these scores to improve models that give educational answers, making them easier to follow and more helpful.
What this means in practice
- •For language model trainers: Build better pre-training data sets by filtering educational texts using multiple quality criteria to improve small model performance on diverse benchmarks.
- •For ai chatbot developers: Enhance chatbots’ educational responses by using multi-criteria quality scores as rewards during post-training to improve pedagogical quality and instruction adherence.
Authors
Oliver G. B. Garrod, Robin A. A. Ince, Meng Liu, Mohamed Huti, Moritz Boos, Amy Waldock, Dominic Andrews, Paul Atherton
Abstract
Educational data filters have become a practical way to improve language-model pre-training, but most filters treat educational value as a single scalar property. This may be too broad for some applications, especially if the data set already features a high density of educational material. Useful learning material needs to be accurate, engaging, well structured, and appropriate for the intended audience and application (e.g. learner- vs teacher-facing). Following QuRating (Wettig et al. 2024), we introduce Edu-QuRating: a pipeline for multi-dimensional educational data scoring and curation. Edu-QuRating defines education-specific rubrics, uses an LLM judge to label sampled document pairs and distills those pairwise preferences into reusable Edu-QuRaters, which can score individual text chunks on a set of educational criteria. Across two sequence-classification base models and six educational criteria, the best Edu-QuRater recovers held-out GPT-4.1-mini pairwise judgements with mean accuracy 0.917. We then apply the resulting scorers in two applications. First, we investigate the potential of Edu-QuRaters for corpus filtering to improve pretraining of small language models. We scored 322.25M FineWeb-Edu-Fortified documents to obtain a filtered pre-training mixture. In matched single-run pre-training comparisons, models trained with Edu-QuRating-based mixtures reached higher observed aggregate accuracy across nine benchmarks than the FineWeb-Edu baseline, with gains concentrated in particular tasks. Second, we used Edu-QuRater scores as reward terms for GRPO post-training. In held-out pairwise judge evaluations, combining Edu-QuRater and answer-structure rewards produced responses preferred to the Qwen3-4B base model on both pedagogical quality and instruction following.