Language model helps create better banking presentations with mixed scores
Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking
Artificial Intelligence
Summary
Making presentations for big bank decisions is tricky because they combine lots of financial facts and advice. The authors studied a system that uses a powerful language model along with financial checks and templates to help make these presentations. Judges scored the presentations from the system higher than simpler versions, but it’s hard to tell if the improvements come from better presentations or from changes in how the judges rate them. The judges also sometimes disagreed on rankings and their scores changed when they rated the same work again, showing it’s tough to measure small improvements reliably.
What this means in practice
- •For banking technology teams: Improve automated generation of financing presentations by integrating language models with financial calculations and validation to support credit decisions.
- •For business intelligence developers: Build tools that produce financial narratives with built-in quality checks to help advisory teams create consistent and traceable presentation decks.
Authors
Ludovic Gibert, Matis Despujols, Andre-Louis Rochet
Abstract
Corporate and investment banking teams use presentations to support credit decisions and advise clients on financing and transactions. Producing these decks requires reconciling financial data, tracing sources and turning analysis into a recommendation. We retrospectively study the development of an agentic harness combining a 27B language model, financial calculations, narrative templates and validation checks. LLM judges guide engineering changes and assess the resulting decks, raising the question of whether higher scores reflect better documents or changes in grading. In shared-session text-only grading with template markers removed, five judges score the complete system 20.4 to 33.6 points out of 95 above the same model generating directly from a short prompt. Every judge scores the system higher on all seventeen development deliverables. Margins against direct Opus generation from a short prompt range from -4.7 to +0.8 points. Judges agree on broad progress across development rounds but agree less on final-deck rankings than on pooled scores. Repeated grading also shifts scores on unchanged decks, making small improvements difficult to distinguish from judge variability.