Large language models show varied confidence across languages and tasks
Beyond Calibration: Temporal LLM Confidence Assessment Across Languages and Local Settings
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) are expected to know how sure they are about their answers. The authors studied two types of confidence: before solving a problem and after solving it. They tested these confidence signals in English and Marathi, a language spoken in India. They found that confidence after solving is usually more accurate, but confidence signals get worse when dealing with local, region-specific knowledge. Also, the models often stay confident even when they make stuff up.
What this means in practice
- •For natural language processing engineers: Improve systems that allocate resources or decide to trust model outputs by using post-solve confidence signals, even in multiple languages.
- •For multilingual chatbot developers: Develop chatbots that better handle regional knowledge by recognizing that confidence signals degrade when local, language-specific facts are involved.
Authors
Sahil Kale
Abstract
Large language models are expected to estimate their own uncertainty, with confidence signals used to guide retrieval, adaptive compute allocation, routing, and selective abstention. We investigate whether confidence signals measured at different temporal stages of problem solving encode distinct forms of epistemic information and remain reliable beyond English-centric settings. Inspired by classical meta-memory theory, we distinguish anticipatory pre-solve confidence (Feeling-of-Knowing, FOK) from retrospective post-solve confidence (Judgement-of-Learning, JOL). We evaluate these signals across two frontier LLMs, multiple reasoning domains, translated Marathi benchmarks, and a newly constructed benchmark of genuinely local Marathi factual questions. Across settings, the cheaper pre-solve signal remains weakly calibrated, while post-solve confidence generally provides stronger epistemic alignment despite its additional inference cost. Confidence behaviour transfers surprisingly well under direct language translation, with calibration trends largely preserved between English and Marathi, but degrades substantially on local Marathi knowledge, suggesting that linguistic transfer alone is insufficient for region-specific information needs. Finally, hallucinated responses maintain persistently elevated confidence after answer generation, highlighting important limitations of current LLM self-assessment mechanisms.