Large language model text scores vary significantly across providers
Same Text, Different Numbers: The Divergence of LLM-Based Measures
Artificial IntelligenceComputation and Language
Summary
Many people use large language models to measure things like sentiment or risk from company texts. This paper shows that different models often give quite different scores for the same texts. These differences affect how results are interpreted and can’t be explained by uncertainty in the texts themselves. The authors suggest that users should treat these scores as dependent on the specific model used and check results with multiple models to be more reliable.
What this means in practice
- •For financial data teams: Use multiple language models to check the stability of text-based financial indicators extracted from earnings call transcripts.
- •For corporate risk analysts: Validate climate and political risk measures derived from text by comparing outputs across several LLM providers to reduce model bias.
Authors
Hamid Boustanifar, Sasan Mansouri
Abstract
Researchers increasingly use generative large language models (LLMs) to convert corporate text into empirical variables. We examine the extent to which LLM-based textual measures are invariant to model choice using thirteen measures, including sentiment, management clarity, uncertainty, answer specificity, and climate and political risk. Seven LLMs from different providers score earnings call transcripts of S&P 500 companies on these constructs. Cross-model rank correlations average only 0.52, and transcript-level differences common across providers account for only 34% of total score variation. Cross-model disagreement does not predict subsequent analyst or market disagreement, consistent with a substantial model-specific component rather than common ambiguity in the underlying disclosure. Model choice significantly affects downstream inference, with coefficient magnitudes, signs, and statistical significance varying substantially across models. Averaging across providers makes transcript rankings more stable for most constructs, but score levels remain sensitive to the models included in the ensemble. LLM-generated variables should therefore be treated as model-contingent measurements and validated across providers.