Large language model tutors show varied fairness across student groups
EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics
Artificial Intelligence
Summary
This work studies how equally large language model tutors help students from different backgrounds. The authors create a testing system that lets a simulated student interact with tutors pretending to be from diverse groups. They find that better model skill doesn’t always mean fairer tutoring, and some types of demographic information create bigger differences in help given. Their method measures fairness using detailed conversation ratings and statistical tests.
What this means in practice
- •For educational technology developers: Evaluate and improve fairness of AI tutoring systems across varied student demographics using standardized benchmarks.
- •For language model trainers: Assess how reinforcement learning and demographic cues impact fairness in teaching quality to design more equitable models.
Authors
Jiaxu Zhao, Bahar Radmehr, Fares Fawzi, Tanya Nazaretsky, Tanja Käser
Abstract
Large language models (LLMs) are increasingly deployed as tutors, but it is unclear whether they support all students equally well. We introduce \textbf{EduFair-Bench}, a benchmark for auditing the pedagogical fairness of LLM tutors---whether tutoring quality varies systematically with student demographics. EduFair-Bench pairs a multi-domain question bank (mathematics, physics, chemistry) with a controlled simulation in which a fixed LLM student interacts with each tutor across nine demographic levels spanning four dimensions: gender, immigration background, first language, and socioeconomic status (SES). Tutoring quality is scored on five turn-level pedagogical metrics and four conversation-level dimensions, using an LLM judge validated against three-annotator consensus on 180 tutor turns. Bias is measured via paired Wilcoxon signed-rank tests and bootstrap effect-size confidence intervals. Two ablations (demographic cues conveyed through names; conflicting demographic information between tutor and student) disentangle tutor-driven from student-driven bias. Across five tutors, we find that model capability and demographic fairness are largely orthogonal: the smallest model is the most consistent while the four more capable tutors all exhibit wide demographic gaps with no clear capability-to-fairness ordering, pedagogy-specific RL training redistributes rather than removes bias, and language- and immigration-related cues produce larger gaps than gender- and SES-related cues.