Jev and language models show different strengths on contract decisions
Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding
Computation and Language
Summary
When computers read contracts to answer questions, they can give the right or wrong answers in different ways. The authors compared a tool called Jev with nine language models on contract understanding tasks. Jev was cheaper and faster but less accurate, while language models were more accurate but costlier and slower. They also found that agreement rates don’t tell the whole story because models can repeat wrong answers or switch answers depending on how questions are asked. This work shows it’s important to check cost, speed, and consistency, not just average accuracy.
What this means in practice
- •For legal technology developers: Optimize contract analysis tools balancing cost, speed, and accuracy for contract question answering.
- •For enterprise ai teams: Select and configure AI models for legal document tasks ensuring consistent and reliable judgments under varied query settings.
Authors
Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng, Lei Fan, Xinhua Ji, Cunyuan Zheng, Huangyong Shan, Philip S. Yu, Xue Liu, Yu Chen, Preslav Nakov, Songwei He
Abstract
Contract inference requires multiple judgments about a shared document, but aggregate accuracy can conceal changes in the individual decisions. Repeated agreement is also insufficient: a model may consistently return the wrong answer. In this paper, we compare Jev with nine language models on ContractNLI, evaluating inference cost, response time, average correctness, and correctness across repeated request conditions. Controlled comparisons vary hypothesis visibility, requested outputs, and output order while keeping the contract and target judgment fixed. Jev has the lowest cost and median response time among the evaluated configurations, while hosted language models achieve higher baseline accuracy. Rankings by baseline accuracy differ from rankings by correctness across every condition and repeat, although small differences in the latter do not establish a general stability advantage. Development diagnostics further reveal compensating corrections and regressions, as well as persistent errors. These findings motivate evaluating cost and response time alongside whether individual judgments remain correct as the request configuration changes. Code: https://github.com/ZF-Utokyo/Jev-Benchmark