Policy revises its own rubrics to improve task performance
RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization
Artificial Intelligence
Summary
Sometimes computer programs that learn by example make mistakes because the rules they were taught don’t cover everything. The authors created a method called RubricOpt that lets a program find and fix gaps in its own training rules. It carefully changes these rules based on where it still fails, helping it get better at tasks without changing the way success is measured during training. Tests showed that this approach can improve program behavior on some tasks, although results vary by case.
What this means in practice
- •For machine learning engineers: Improve AI models by enabling them to adjust their own training criteria to address repeated failures without changing evaluation metrics.
- •For automated content generation teams: Refine content generation behaviors by integrating semantic rubrics that models can self-edit to better align with task requirements.
Authors
Wei Chen, Yubing Wu, Shigui Li, Delu Zeng, Jingfeng Liu, Junmei Yang, Zhiheng Zhou, Bo Li, Min Chen, Zhou Wang
Abstract
A model's failures can reveal gaps in the behavioral criteria guiding its training. We introduce RubricOpt, a framework that lets a policy revise these criteria within preference optimization. The policy diagnoses its own rubric-free outputs, retains requirements already covered by the rubric, and edits clauses that leave recurring failures unaddressed. A shared semantic representation connects task demands, preference-data contrasts, and rubric clauses, while a deterministic gate checks evidence references and fixed semantic anchors. The rubric then guides Direct Preference Optimization (DPO) during training, with preference pairs, labels, and the objective held fixed, and is removed at evaluation. Our analysis relates the local effect on rubric-free success to semantic alignment and a residual beyond the measured factors. Experiments on capability and open-ended generation benchmarks show that rubric conditioning can improve outcomes over ordinary DPO, with gains and regressions varying across models and datasets. A matched continuation isolates the contribution of revision from additional training and finds mixed incremental effects for the tested edit, whose feedback overlaps the evaluation set. RubricOpt thus brings the behavioral specification itself into the policy's feedback loop.