Jev offers cost-effective multilingual text judging with some limits
Is Jev a Good Multilingual Judge?
Computation and Language
Summary
Deciding if a piece of text is good or bad in many languages is hard and usually expensive with big AI models. The authors tested a model called Jev, which gives a probability about the correctness of text, to see if it can judge well across many languages. Jev works well and costs much less than a bigger model called GPT-6.1, but it struggles when checking detailed knowledge answers, especially in less common languages. The authors also suggest ways to better use Jev by looking at its confidence numbers and choosing when to ask it to judge.
What this means in practice
- •For machine translation teams: Use Jev to cheaply assess translation quality across many languages, balancing cost and accuracy especially for common languages.
- •For customer support automation teams: Integrate Jev to automatically verify multilingual AI-generated answers with confidence scores to prioritize human review when needed.
Authors
Wei Liu, Yi Fan, Xiyan Fu
Abstract
Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately $1/31$ to $1/61$ of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.