Propalign estimates proposition importance from multiple summaries
The Sum of All Summaries: PROPALIGN for Graded Proposition Salience Evaluation
Computation and Language
Summary
Figuring out which ideas or points in a document matter most is important for many AI tasks, but it is often unclear how to score these points consistently. The authors introduce PROPALIGN, a method that uses multiple summaries of documents to better estimate how important each proposition is. They also created a large dataset called PROPALIGNBANK with many examples of aligned propositions and summaries. Their tests show that asking AI models directly can lead to inconsistent results, while their summary-based method is more stable and can approach human-level accuracy when multiple models work together.
What this means in practice
- •For document summarization teams: Improve ranking of key ideas in documents by using multi-summary alignment for more stable proposition importance scoring.
- •For legal technology developers: Enhance extraction of salient legal points from cases and contracts using graded proposition salience scoring based on multiple summaries.
Authors
Lin Ai, Emma Thronson, Katherine Conhaim, Amir Zeldes
Abstract
Identification and ranking of salient propositions in documents is indirectly involved in many AI tasks, but few papers define proposition salience scores clearly, make diverse data available, or evaluate models directly on the task. This paper proposes PROPALIGN, a multiple-summary-based approach for propositional salience score estimation, and releases PROPALIGNBANK, an accompanying dataset for the method with 138K alignments of over 35K propositions. We compare several approaches to salient proposition extraction in English across a broad range of spoken and written genres, and demonstrate that naively asking models for judgments leads to inconsistencies. By contrast, graded, summary-based metrics are more stable. We further observe that while single models perform worse than humans, an ensemble of models incorporating linguistic features approaches manual annotation quality ceilings. In addition, analyses of the model outputs suggest insights into how models can extract salient information with and without the use of summaries.