AI summaryⓘ
The authors explain that language models give probabilities for word choices, but real professional decisions need reliable uncertainty about meaningful states like diagnoses or market conditions. They propose a method called a semantic map to convert probabilities over words into trustworthy probabilities over these important states. This method uses a reference model, calibration data, and conditions to ensure stable, accurate results that don’t change with different wordings of the same information. Their tests with financial texts show this approach gives better and more stable uncertainty estimates than just using the model’s raw word probabilities. This work helps turn language model outputs into scientifically valid, auditable probabilities useful in professional settings.
language modelssemantic mapposterior probabilitiescalibrationlatent stateBayesian updatinguncertainty quantificationprompt engineeringstabilityauditing
Abstract
Language models produce probabilities over words, but professional decisions require uncertainty over meaningful states such as diagnoses, hypotheses or operational conditions. A model's printed numerical confidence does not establish reliability. We introduce a semantic map: a prespecified, testable bridge from probabilities over verbal responses to probabilities over declared states, formulated as semiparametric inference for a finite-valued latent state. A reference model defines the target posterior, the language model supplies an unrestricted conditional distribution over verbal responses, and held-out calibration connects them. We derive posterior-error bounds and conditions for existence, uniqueness, stability and sequential Bayesian updating. Crucially, language probabilities depend on the prompt's lexical form, whereas the target posterior is unchanged by information-equivalent rewording. We test the method on professional market text compiled from Federal Reserve economic and financial series and on controlled simulations with exact posteriors. Across two fitted language models, language-derived probabilities outperform printed numerical probabilities, recover held-out posteriors with valid uncertainty coverage, remain largely stable under paraphrasing, and respond appropriately to altered evidence. The broader implication is that prompt engineering optimizes a wording-dependent response, whereas scientific and professional use requires validated stability of application-relevant meaning. The semantic map turns this general concern into a testable statistical problem and, when its acceptance conditions hold, yields an auditable posterior estimate. The same principle offers a template for auditing classifications, recommendations and other fluent responses that may conceal semantic instability.