Process aware method improves math problem quality assessment
ProIQA: A Process-Based Framework for Fine-Grained Math Item Quality Assessment
Artificial Intelligence
Summary
Assessing the quality of automatically created math problems is challenging because current methods don't fully capture the steps needed to solve them. The authors propose a new approach called ProIQA that looks at how a problem is solved by building detailed reasoning trees from solutions. This approach uses advanced AI models to understand both the problem text and the solving steps together. Their tests show that ProIQA can better evaluate math problems’ educational value, helping systems that create personalized math exercises.
What this means in practice
- •For online learning platforms: Improve the evaluation of automatically generated math exercises by using process-aware assessments that consider problem-solving steps.
- •For educational content creators: Use process-based quality metrics to better select and adapt math problems for various student skill levels and learning objectives.
- •For adaptive testing system developers: Incorporate solving-process representations to enhance automatic difficulty and knowledge concept assessment in math tests.$Commercial implications: This paper enables commercial adaptive math testing tools to automatically score and select items with fine-grained quality and difficulty estimates.
Authors
Junkai Tong, Mingjia Li, Haoran Chen, Yaoyu Jiang, Hanjie Ge, Yixuan Wang, Hong Qian
Abstract
Automatic Item Generation (AIG) is pivotal for personalized education, yet guaranteeing the pedagogical value of generated items remains a bottleneck. Existing Item Quality Assessment (IQA) methods typically rely on unscalable manual reviews or shallow stem-based metrics, failing to capture the reasoning process required for mathematical problem-solving. To bridge this gap, this paper proposes Process-based Item Quality Assessment (ProIQA), a process-aware framework for fine-grained quality assessment of math items. We first formulate IQA across three heterogeneous dimensions, including knowledge concepts, difficulty, and disciplinary competencies, under a unified process-aware perspective. Based on this formulation, we construct a process-enhanced IQA resource by augmenting original item data with structured reasoning trees derived from raw solutions. Technically, ProIQA leverages Large Language Modelsto construct hierarchical reasoning trees and employs Graph Neural Networks (GNN) to encode their topological dependencies and procedural semantics. The resulting solving representation is fused with stem semantics through a dual-view (``Stem + Solving'') architecture, enabling comprehensive assessment across learning objectives. Extensive experiments on K12 mathematical datasets show that ProIQA effectively captures process-oriented features, offering a scalable data-driven solution for evaluating AIG outputs in intelligent education systems.