Llm accuracy varies on middle and high school science benchmarks
A Benchmark for LLM's Understanding of Middle School and High School Science Topics
Artificial IntelligenceComputers and SocietyHuman-Computer Interaction
Summary
Large language models (LLMs) are being used more in schools, but teachers don’t have good tools to check how well they understand school science subjects. The authors created a special test aligned with school science standards to see how nine different LLMs perform on middle and high school science topics. They found that bigger models aren’t always better and some smaller ones worked well. Human experts also checked the test questions and found that LLM-generated questions don’t always perfectly match science standards, showing the value of humans in creating teaching materials.
What this means in practice
- •For educational software developers: Use the benchmark to select or improve LLMs for science tutoring applications targeting middle and high school curricula.
- •For edtech curriculum teams: Incorporate human review with LLM-generated content to better align interactive science materials with established educational standards.
Authors
Noah L. Schroeder, Yessy Eka Ambarwati, Yuji Zhang, ChengXiang Zhai
Abstract
Large language models (LLMs) are increasingly integrated into educational settings, yet educators lack robust, standards-aligned tools to evaluate their effectiveness in K-12 science contexts. Existing benchmarks predominantly assess general language or advanced scientific reasoning, leaving a critical gap in understanding LLMs' performance on content directly relevant to secondary science curricula. To address this gap, we developed a comprehensive NGSS-aligned benchmark for both middle and high school science using a rigorous synthetic data pipeline, multi-judge validation, and item-level psychometric analysis. Nine open-weight LLMs were systematically evaluated using this benchmark, indicating that several smaller, locally deployable models achieved high accuracy across diverse science domains and question types. Our findings indicate that model size did not consistently predict performance, emphasizing the importance of intentional model selection for educational deployment. We then incorporated a human reviewer into the loop, reviewing the items generated by the LLMs for alignment with NGSS standards. The human review indicated that synthetically generated items were not in perfect alignment with the NGSS standards, indicating the benefits of human-in-the-loop item development, the need to explore the intersection of content and pedagogical knowledge, and the need to extend benchmarks to evaluate LLMs' capacity for interactive, evidence-based feedback in educational scenarios.