Comparison of metrics for summarizing long financial reports with transformers

A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers

Computation and Language

Summary

Long financial reports from UK companies are very detailed and hard to read quickly. The authors studied different ways to make short summaries using advanced language AI models called transformers. They also looked at various scoring methods to see which best measure how good the summaries are. They introduced a new combined score, BRUGEscore, that mixes two popular scores to better capture summary quality. Finally, they tested if their results are reliable and checked how the methods perform when the input reports are deliberately changed.

What this means in practice

  • For financial data analysts: Use the combined BRUGEscore to better evaluate AI summaries of long company reports for investment decisions.
  • For legal document reviewers: Employ transformer-based summarization with robust evaluation metrics to condense lengthy legal documents for faster review.

Authors

Nadhem Zmandar, Mo El-Haj, Paul Rayson

Abstract

There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year. UK annual reports are very lengthy documents with around 80 pages on average. In this study, we aim to benchmark a variety of summarisation methods on a set of different pre-trained transformers with different extraction techniques. In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries. We hypothesise that some evaluation metrics do not reflect true summarisation ability and propose a novel BRUGEscore metric, as the harmonic mean of ROUGE-2 and BERTscore. Finally, we perform a statistical significance test on our results to verify whether they are statistically robust, alongside an adversarial analysis task with three different corruption methods.