Dialogue compression evaluation reveals key information loss points
TPBench: A Turning-Point Benchmark for Dialogue Compression
Computation and LanguageArtificial Intelligence
Summary
When computer programs shorten conversations, they can accidentally drop important updates that change earlier information, like correcting a price or revising a choice. The authors show that standard scores don't catch these mistakes well because they mix original and updated information. They introduce TPBench, a new test that checks if the shortened conversation keeps both the user's starting request and any changes made later. Their tests reveal that removing the turns where updates happen causes accuracy to drop, highlighting the importance of preserving these 'turning points.'
What this means in practice
- •For chatbot developers: Evaluate and improve dialogue compression methods to preserve essential user corrections and updates during conversations.
- •For customer service platform teams: Enhance compressed conversation summaries to retain critical changes in user requests, reducing misunderstandings in automated response systems.$Commercial implications: Enables more accurate compressed dialogue summaries for customer support software, improving service quality and user satisfaction.
Authors
Minji Park, Seunghyun Yoon, Hyuk Lim
Abstract
A compressor can keep the facts of a dialogue and still drop the turn that changed them. A user corrects a price, reverses a choice, or adds a constraint. We call this failure turning-point eviction. One overall retention score hides it, because that score mixes what the user first wanted with what the user wants now. We introduce TPBench, which evaluates three complementary information targets at shared nominal retention budgets. P1 asks for the user's initial goal. P2 asks for the current value of a slot the user revised. P3 asks for both, in dialogues with a late annotated slot update. The current-value answers come from the human dialogue-state annotations of MultiWOZ and SGD. The initial-goal answer is the first sentence of the first user turn. Neither requires new crowdsourcing. The probe-specific evaluations rank compression methods differently. On the joint probe at a retained fraction of 0.30, every tested compressed method remains below full context with the main Llama reader. Deleting the turn that carries the update sharply lowers current-value accuracy, while deleting one matched irrelevant turn leaves it unchanged. A Mistral reader repeats the P2/P3 rankings and the joint-probe gap. Current-value recovery is tested on an additional corpus, LongMemEval-KU, and on Chinese RiSAWOZ: full context has the highest accuracy, and recency has the highest compressed-method mean in both evaluations.