Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
Computation and Language
Summary
The authors study how to create useful rubrics—rules to judge answers—for training large language models. They propose Rubrics on Trial, a method that starts with no rubrics and gradually builds good ones using only the questions, without needing human input or training other models. Their system tests each new rubric to make sure it can fairly tell good answers from bad and avoids rubrics that are too narrow or only focus on style. Their experiments show this approach works well across multiple tests, beating existing methods in most cases.
Authors
Haocheng Yang, Licheng Pan, Xiaoxi Li, Zhichao Chen, Zhiheng Zhang, Yuan Lu, Haoxuan Li, Hao Wang
Abstract
Rubrics provide structured, fine-grained signals for training and evaluating large language models (LLMs). Yet reliable query-specific rubrics are difficult to construct. Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses. Direct query-to-rubric generation avoids these resources, but provides no explicit check that a plausible rubric is useful. Such a rubric may fail to distinguish answer quality, reward an optional style, or penalize a valid alternative strategy. We introduce Rubrics on Trial, a query-only framework that evolves a rubric set from an empty set without external annotations or model training. It derives supervision solely from synthetic rubric-conditioned response pairs and validates each proposed rubric before adding it, screening out non-discriminative, over-specific, and style-only candidate rubrics. Experiments across five preference benchmark suites demonstrate the effectiveness of Rubrics on Trial, which achieves the best average accuracy and leads on six of seven evaluation sets.