Semantic scaffold method improves summary evaluation beyond current metrics
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Computation and LanguageArtificial Intelligence
Summary
Measuring how good a summary is can be tricky because common tools either only check for word overlap or give nearly identical scores for very different summaries. The authors noticed this problem happens in many different data sets and languages. They created a new way called Semantic Scaffold that breaks down the important facts, questions, and details in the original text and checks if these key parts are kept in the summary. Their new scores can better tell which summaries keep the important information without overwhelming detail, even when old methods fail.
What this means in practice
- •For software engineers: Improve automatic selection of summarization models by using semantic scaffold scores that better rank summaries based on essential content.
- •For multilingual app developers: Use semantic scaffold evaluation in apps producing summaries across languages to maintain important information when conventional scores saturate.
Authors
Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger, Sepideh Kharaghani, Ying Zhang
Abstract
Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a source text, labeling each as a main point or supporting detail, and reusing this structure as a fixed reference for scoring summaries. From this representation, we derive three diagnostic metrics: Fact Preservation Score (FPS), Question Preservation Score (QPS), and Entity Preservation Score (EPS), designed to reward the preservation of essential information while penalizing detail overload, and position them as interpretable diagnostics that remain informative where holistic axes collapse. Finally, we analyze four recurring failure modes of ROUGE and LLM-as-judge scores, demonstrating that scaffold-based evaluation remains informative where conventional metrics collapse.