Decisiontracebench tracks evolving critical info in multi-person speeches
DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations
Computation and Language
Summary
In conversations where important decisions are made, like in finance, understanding how the information changes and who said what is really important. The authors created DecisionTraceBench, a new way to test if computers can follow these evolving decision details across multiple speakers. They found that current systems struggle to keep track of the full story, especially when identifying who said each part and when it happened. Their work provides a new challenge to improve speech understanding in important, high-stakes discussions.
What this means in practice
- •For financial compliance teams: Track and verify evolving crucial information and speaker contributions in financial spoken dialogues for audit and regulatory review.
- •For call center quality analysts: Evaluate how well automated systems understand and trace important decision updates and attribution during multi-agent customer calls.
Authors
Wenchao Du, Fangxin Shang, Long Wu, Runze Cui, Qing Yang, Yehui Yang
Abstract
In high-stakes domains such as finance, critical information evolves through multi-party interaction. Recovering these changes and their sources is fundamental to reliable speech understanding. Existing speech benchmarks focus on transcription, local understanding, and turn-level state prediction, leaving joint decision evolution across speakers insufficiently evaluated. We introduce DecisionTraceBench, a speech state-tracking benchmark for Decision-Critical Information State Tracking (DCIST). It jointly evaluates decision event sequences, speaker attribution, scope, and temporal evidence, using trajectory-level F1 to measure complete event-chain recovery. Our dataset comprises Chinese financial dialogues involving two to four speakers, with 100.39 hours of synthetic audio and 8.61 hours of human-performed recordings. On a fixed cohort of human-performed recordings, five cascaded systems and two direct-audio models achieve DCIST scores of 5.73% to 21.35%. Ablations with speaker-labeled reference scripts and no temporal localization requirements yield a maximum score of only 38.72%. Audio perturbation experiments further show that overall transcription error rates do not adequately reflect complete trajectory recovery. These findings reveal limitations in jointly understanding information provenance and state evolution, providing a benchmark for evaluating decision-process understanding and decision traceability in high-stakes speech applications.