Large language models evaluated for trustworthy financial transcript analysis
A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Computation and LanguageArtificial Intelligence
Summary
Financial earnings call transcripts are long and complex, making it hard to check if computer-generated summaries are accurate and can be trusted. The authors created a new way to measure how well these language models base their answers directly on the actual transcript, without needing expensive expert reviews. They also built a dataset of 100 real company transcripts to test these models. The study found that while models usually cite evidence correctly, they often struggle to give fully accurate answers and sometimes can’t tell when they don’t have enough information to respond.
What this means in practice
- •For financial analyst teams: Improve automated tools to verify if financial report summaries use accurate evidence from transcripts.
- •For legal compliance teams: Detect when automated financial analyses may lack sufficient backing to avoid unsupported claims in reports.
Authors
Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo, Li Zhang, Zheng Ma
Abstract
Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations. Empirical results show that LLMs perform well in groundedness but face notable limitations in correctness, with informational insufficiency presenting an additional challenge.