Pinocchio estimates confidence on black-box language model answers
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Artificial Intelligence
Summary
People using large language AI for important decisions want to know not just the answers, but how sure the AI is about those answers. The authors created Pinocchio, a tool that guesses how likely a language model’s answer is correct without needing to see inside the model or adjust it. It works well even with models it wasn’t trained on, and gives quick uncertainty scores by just looking at the text output. This lets users better trust or double-check AI responses when the underlying AI is a closed system.
What this means in practice
- •For ai product developers: Add reliable confidence scores to outputs of closed-source API language models without model access or tuning.
- •For finance risk teams: Assess language model answer confidence in financial decision systems using black-box APIs to reduce risk.
Authors
Kevin David Hayes, Arka Pal, Haosong Zhang, Tom Goldstein, Micah Goldblum
Abstract
In high-stakes decision-making applications of large language models (LLMs), practitioners require not only accurate LLMs but also uncertainty estimates for their predictions. Existing approaches to uncertainty estimation for LLMs require access to log-probabilities output by the model or require fine-tuning access. However, many industrial LLM products use closed-source API models, and many such API models like GPT do not return log-probabilities and may not allow fine-tuning. We introduce Pinocchio, an external calibrator that estimates the correctness of responses from black-box API models. Trained jointly on responses from seven LLMs, it achieves 0.862 AUROC predicting the correctness of held-out responses from those same models, and shows zero-shot transfer to thirteen unseen models across eight organizations. Our model needs only a single forward pass to generate an uncertainty estimate and requires no access to the target model's logits, weights, or internal states. A lightweight text only 0.8B checkpoint matches our largest model's AUROC. We release code for adding uncertainty estimation to existing repos in only two additional lines of code.