Legal reward framework improves language model responses across multiple quality dimensions
LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models
Computation and Language
Summary
Legal language models often struggle to evaluate their answers beyond just being right or wrong. The authors created LexReward, a system that judges legal responses based on style, content elements, and logical reasoning. This detailed scoring helps the models learn to give clearer, more accurate, and well-reasoned answers. Testing showed that using this system improves how well the models perform in legal tasks without needing exact correct answers for comparison.
What this means in practice
- •For legal ai developers: Improve legal language model training by providing structured, dimension-specific reward signals for better legal answer quality.
- •For legal tech companies: Enhance AI-powered legal document and response generation by optimizing models through detailed reward feedback on style, content, and reasoning.$Commercial implications: Enables development of AI legal assistants that deliver higher-quality, domain-specific responses for legal service providers.
Authors
Yida Cai, Xin Dai, Bingxiang He, Huiyuan Xie, Yuxiao Ye, Zhenghao Liu, Yang Bai, Zhiyuan Liu
Abstract
Legal language models require reward signals that capture not only answer correctness but also the multidimensional quality of legal responses. Existing reward methods, however, often rely on coarse-grained holistic judgments, providing limited domain specificity and interpretability. We introduce LexReward, a taxonomy-driven framework for legal reward modeling. LexReward characterizes legal response quality along three complementary dimensions: Style, covering lexical and syntactic quality; Element, assessing legal subjects, facts, statutes, and decisions; and Chain, evaluating the order, completeness, correctness, and non-redundancy of legal reasoning. For each dimension, we develop rubrics that specify evaluation criteria and quality levels. The resulting rewards are used to construct pairwise preference data for Direct Preference Optimization (DPO) and reward-model training. Experiments show that the rubric-based rewards reliably distinguish legal responses of different quality and that DPO training on the preference data improves performance across all three dimensions. The learned reward models, LexRM, also support effective downstream optimization: each dimension-specific reward model improves policy performance in its corresponding dimension through reinforcement learning, without requiring reference answers at reward time. Dimension-wise analyses further support the effectiveness of the proposed taxonomy and reward construction.