Evidence centered design improves prompts for automated essay scoring
Using Evidence-Centered Design to Guide Prompt Engineering in Measurement: An Illustration with Automated Essay Scoring
Artificial IntelligenceHuman-Computer Interaction
Summary
Automated essay scoring uses AI to grade essays, but how you ask the AI matters a lot. The authors show that by using a structured process called Evidence-Centered Design, prompts can be designed more carefully to make AI scoring fairer, more reliable, and more accurate. This replaces guesswork in prompt writing with clear steps based on educational measurement principles. Their study found this structured way makes AI scoring results better than usual prompt methods.
What this means in practice
- •For educational technology developers: Design prompts for AI scoring systems with structured guidelines to ensure fair and valid grading of student essays.
- •For language assessment teams: Implement a principled workflow for developing prompts that enhances the reliability and fairness of automated language tests.
Authors
Jiawei Xiong, Cheng Tang, Hong Jiao, Hanna Choi
Abstract
Generative artificial intelligence (GenAI) models, particularly large language models (LLMs), are increasingly used in educational settings for tasks such as item generation, automated scoring, and feedback delivery. The quality of these outputs depends not only on the AI model but also on how users construct the prompts that guide model behaviors. Current prompt engineering practices are largely ad hoc, relying on trial and error rather than principled design. This lack of structure introduces validity threats including construct misrepresentation, construct-irrelevant variance, and limited reproducibility. We propose that Evidence-Centered Design (ECD), a framework in educational measurement for structuring valid inferences from observed performance, can serve as a principled foundation for prompt engineering in educational applications. We map the three core ECD models, the student model, the evidence model, and the task model, onto the components of educational prompt design, and describe a workflow for ECD-based prompt engineering. We derive five design principles from this mapping and present an empirical study contrasting ECD-designed prompts with state-of-the-art (SOTA) prompting baselines on an automated scoring task. Our empirical study demonstrates the usefulness of this framework on automated scoring. Results suggest that treating prompting as assessment design rather than informal instructions strengthens the validity, reliability, and fairness of AI-generated educational outputs.