Legal judgment prediction improved by evolving queries and rubrics
LegalEvolver: Agentic Legal Judgment Prediction via Rubric-Guided Self-Evolution
Artificial IntelligenceComputation and Language
Summary
Legal judgment prediction is hard because it needs models to find important legal information and think carefully step-by-step. The authors created LegalEvolver, a tool that makes better search questions by improving them over time, guided by special evaluation rules called rubrics. This process helps the model get better at finding and using information, going beyond its initial skills. Tests show this approach works better than standard methods for predicting legal outcomes.
What this means in practice
- •For legal tech developers: Create tools that refine search queries automatically to improve legal case outcome predictions.
- •For document retrieval engineers: Design retrieval systems that iteratively enhance query quality using rubric-guided feedback.
Authors
Xinyu Yang, Jiejun Tan, Chenlong Deng, Chenghao Zhang, Tongyu Wen, Zhicheng Dou
Abstract
As LLM-based agents develop, they have been increasingly applied in the legal domain. Legal Judgment Prediction~(LJP) requires models to conduct multi-step reasoning and external legal evidence retrieval. Existing legal search agents' performance is often constrained by the quality of retrieval queries. They often formulate queries directly from the initial user request, overlooking latent legal issues and legally salient details. To address this issue, we propose \emph{LegalEvolver}, a query optimization framework specifically designed to generate legally informative queries for LJP-oriented retrieval. Sampling high-quality optimized queries and their corresponding search trajectories is challenging, as relying solely on unguided high-temperature rollouts is both inefficient and constrained by the model's initial capabilities. We therefore heuristically optimize queries as seeds, improving the efficiency of trajectory generation. Furthermore, we summarize rollout experience into rubrics and use them to guide multi-round iterative self-evolution. Iteratively performing rubric optimization and rubric-guided RL enables the model to surpass its initial performance ceiling. End-to-end performance on extensive benchmarks demonstrates the effectiveness of query optimization on LJP. Comparisons with direct SFT followed by RL demonstrate the effectiveness of rubric guidance. Evolving rubrics further outperform fixed rubrics, confirming the benefits of self-evolution.