Open models extract accounting relationships from financial reports
FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing
Machine Learning
Summary
Financial reports often describe complex accounting relationships in plain text, which can be hard to analyze automatically. The authors study whether openly available large language models (LLMs) can find and label these accounting connections based on a financial graph. They tested several models on thousands of reports and created a new dataset to check how well models classify these relationships. Their results show that while some open models perform well, proprietary models still do better, and different accounting categories vary in difficulty. This work highlights ways to better use language models for structured financial analysis while respecting privacy.
What this means in practice
- •For financial analysts: Automatically extract transaction relationships from financial reports for deeper data insights without manual tagging.
- •For compliance teams: Use locally deployable models to analyze financial texts securely, supporting privacy-sensitive regulatory auditing.
Authors
Ashman Mehra, Bryan Routledge, Jane JaeYeon Pyo, Pierre Jinghong Liang
Abstract
Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predominantly emphasize question answering and classification, schema-guided evidence discovery remains relatively under-explored. We investigate whether open-weight LLMs can extract and classify evidence corresponding to predefined double-entry accounting relationships underlying the text. Using a financial accounting graph comprising 11 nodes (accounts) and 11 edges (transaction types), we evaluate mid-sized models from 5 open-weight families across 1,300 SEC 10-K Item 7 filings spanning 26 years, comparing their extraction performance against three proprietary models. Results reveal substantial differences in extraction volume, edge distributions, and cross-model agreement. Building on this study, we introduce a human-annotated relation classification benchmark comprising 483 passages across three financial accounting categories, evaluating 34 open-weight and 10 proprietary models. The strongest open-weight and proprietary models achieve 84.1\% and 92.8\% accuracy, respectively, with considerable variation across accounting categories. Our findings highlight the importance of evaluating evidence discovery and relation understanding separately, while investigating locally deployable models for scalable, privacy-sensitive financial analysis.