RubricWorld improves large language model grading accuracy with guided rubric edits
RubricWorld: World Model-Guided Rubric Optimization for LLM Graders
Artificial Intelligence
Summary
Grading answers using large language models (LLMs) relies on detailed scoring instructions called rubrics. The authors study how to improve these rubrics without changing the LLM itself by making smart changes to the instructions. Their system, RubricWorld, learns which rubric changes fix grading errors without causing new ones. This approach saves effort by predicting which edits are best to try next, improving grading accuracy significantly in tests.
What this means in practice
- •For assessment designers: Improve automated grading accuracy by refining rubric instructions using RubricWorld to better detect and correct grading errors without changing the grading model.
- •For edtech product teams: Deploy rubric-guided grading systems that adapt scoring instructions based on prior performance to provide more reliable automated scoring in educational platforms.$Commercial implications: This enables adaptive grading products that can sell more accurate automated assessment services to educational organizations.
Authors
Hongji Pu, Hang Li, Yucheng Chu, Jiliang Tang
Abstract
Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.