AI tutoring platform shows promising gains in science exam scores
Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science
Human-Computer Interaction
Summary
Education technology changes quickly, making it hard to know what really works. The authors tested a new AI tutoring tool called Medly with nearly a thousand high school science students over four weeks. Students who used Medly improved their test scores more than those who studied on their own. The tool helped equally across biology, chemistry, and physics, and worked for students of different backgrounds. While results look good, the study was short and missing some data, so more testing is needed.
What this means in practice
- •For education technology developers: Run rapid, smaller-scale trials led by practitioners to evaluate AI tutoring tools during ongoing development and update.
- •For school technology coordinators: Adopt AI tutoring platforms like Medly for GCSE science revision as a supplement to self-study based on initial positive impact.$Commercial implications: Medly can be marketed as an AI tutoring service improving science attainment in secondary schools, backed by preliminary trial evidence.
Authors
Wayne Harrison, Rahil Khowaja, Emma Dobson, Germaine Uwimpuhwe, Steve Higgins
Abstract
Artificial intelligence (AI) systems in education are developing on timescales that sit uneasily with conventional evaluation. By the time a large-scale trial has been designed, delivered, analysed and published, the technology under study may have changed materially. This creates a temporal problem for evidence-informed education: the need for timely evidence can encourage reliance on weak observational or usage data, while conventional rigorous evaluation may produce evidence too slowly to guide rapidly evolving practice. We examine teacher-led micro-randomised controlled trials (micro-RCTs) as one response to this problem. The empirical case is a four-week multisite individually randomised evaluation of Medly, an AI-powered tutoring platform, in GCSE Biology, Chemistry and Physics in English secondary schools. Of 929 students completing baseline assessment, 644 completed post-testing. In the primary ITT analysis, students allocated to Medly achieved higher post-test attainment than students undertaking business-as-usual self-directed revision (Hedges' g = 0.33, 95% CI 0.18 to 0.48). Positive estimates were observed in Physics (g = 0.31), Chemistry (g = 0.32) and Biology (g = 0.52), with no evidence of differential impact by disadvantage status. Greater platform engagement was associated with higher attainment, but these post-randomisation analyses are treated as exploratory rather than causal. Attrition was substantial (30.7%), outcome measures were curriculum-aligned rather than standardised, and process evaluation response was limited. We therefore interpret the findings as preliminary. We argue that the value of micro-RCTs for educational AI lies not in replacing definitive evaluation with small studies, but in enabling a rapid, cumulative evaluation architecture in which randomised estimates can be generated, replicated and updated as technologies and their implementation evolve.