Mubric improves evaluating large language model answers with mutation testing
Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation
Artificial IntelligenceSoftware Engineering
Summary
Evaluating the quality of AI-generated responses can be tricky because it's hard to create good scoring guides automatically. The authors propose Mubric, a method that uses mutation testing—a way to check test quality by introducing faults—to improve rubrics for scoring AI answers. Mubric learns common mistakes from real examples and uses them to test if the rubric properly penalizes these errors. Their tests show Mubric works better than previous automated methods across many tasks.
What this means in practice
- •For ai development teams: Generate more reliable scoring rubrics to better assess AI-generated responses on diverse tasks.
- •For software testing teams: Adopt mutation testing concepts from software to improve evaluation frameworks for AI output quality.
Authors
Jiayuxuan Yang, Jie M. Zhang, Yiling Lou, Zhenpeng Chen
Abstract
Rubric-based evaluation is widely used to assess LLM-based systems by decomposing response quality into task-specific scoring criteria. However, automatically generating rubrics that reliably capture task-specific quality requirements remains challenging. We introduce Mubric, a mutation testing-guided approach to rubric generation. Mutation testing, a classic software testing methodology, evaluates a test suite by injecting faults into programs and checking whether the tests detect them. We draw an analogy between test suites and rubrics: if a rubric captures an important quality requirement, introducing a corresponding defect into an otherwise high-quality response should reduce its score. Mubric first mines common defects from real pairs of preferred and dispreferred responses and abstracts these defects into reusable mutation operators, each specifying how to introduce a particular type of response defect. For a new task, it applies relevant operators to a reference response, checks whether the injected defects reduce response quality, and uses insufficiently penalized defects to refine the rubric. We evaluate Mubric on 703 tasks across four representative domains against six advanced rubric generation methods. Mubric achieves the highest overall evaluation accuracy, outperforming the strongest baseline by 7.48 percentage points.