Languages impact accuracy in multilingual ai confidence checks
Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation
Computation and Language
Summary
When an AI model evaluates whether its answer is correct, three different languages are involved: the question's language, the answer's language, and the language of the instruction telling the AI to judge its answer. The authors studied how these three languages affect AI confidence scores by testing different combinations across six languages. They found that which language matters most depends on the type of test, and that changing the verifier language changes the AI's confidence but not accuracy. This means that to fairly compare AI models on multilingual confidence, all three languages and the evaluation method must be clearly reported.
What this means in practice
- •For machine learning engineers: Build AI systems that better evaluate their answers by carefully selecting languages for questions, responses, and verifiers to improve reliability.
- •For multilingual ai service teams: Ensure consistent model comparisons by tracking all involved languages during confidence scoring to avoid misleading performance gaps across languages.
Authors
Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
Abstract
When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, and that of the verifier instruction asking for the judgment. Multilingual confidence evaluations seldom separate the three. We vary them independently in a six-language factorial design with ten open-weight models (3B-14B) on LSQA, MMMLU and TriviaQA, holding each answer fixed across verifier languages and measuring confidence with a first-token P(True) probe. The three languages move different parts of the evaluation: which of them governs accuracy depends on the benchmark, matching the response language to the question or verifier language sharpens discrimination but not calibration, and the verifier language alone shifts calibration error by up to 0.04 without changing accuracy. The scorer matters as well: an unvalidated judge would have shrunk the MMMLU English-Hindi accuracy gap from 0.198 to 0.038. Multilingual confidence results, model rankings included, are therefore comparable only when all three languages are reported and the scorer is validated.