Open source language models vary in accuracy for ESG report analysis
Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain
Artificial Intelligence
Summary
ESG reports help track how companies manage environmental and social responsibility, but reading many reports is time-consuming. The authors tested several open-source large language models to see how well they can find and generate key ESG information from real company reports using a retrieval method. They found that while these models are good at finding relevant info, their ability to produce accurate and faithful answers differs a lot. The results suggest these models need more fine-tuning for ESG topics to be reliable in this important area.
What this means in practice
- •For corporate compliance teams: Use open-source models to automate extraction of ESG metrics from regulatory reports, improving monitoring efficiency despite current accuracy limits.
- •For financial analysts: Employ evaluated models to support ESG risk assessment by quickly summarizing relevant corporate disclosures from past years’ reports.
Authors
Motaz Saad, Anna Borrelli, Ivan Gentile, Kianna Kazemi, Francesco Piccialli, Antonella Longo
Abstract
Environmental, Social, and Governance (ESG) reporting is critical for corporate accountability, with Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) offering strong potential to automate KPI extraction. However, open-source LLM performance in domain-specific ESG tasks remains insufficiently understood. This paper evaluates open-source LLMs in ESG contexts using a structured framework and evaluation resource based on 498 real-world ESG reports from EU-listed companies (2010-2024). We evaluate seven open-source models (2B to 30B parameters) -- glm-4.7-flash, nemotron-3-nano:4b, qwen3:4b-instruct, gemma3:4b, gemma4:e4b, gemma4:e2b, and ministral-3:8b -- using 100 persona-based synthetic QA pairs covering ESG information needs. System performance is assessed via RAGAS metrics, including contextual recall, precision, relevance, faithfulness, answer relevancy, and factual correctness. Results show notable performance variations across architectures. Retrieval performance is strong across models (context recall around 0.58-0.61, context precision around 0.78-0.81, context relevance 0.965-0.985). Generation diverges most on faithfulness (0.607-0.822) and least on answer relevancy (0.760-0.881): glm-4.7-flash leads in faithfulness (0.822), qwen3 in factual correctness (0.449), and ministral-3 in answer relevancy (0.881). Low overall factual correctness (0.387-0.449) highlights the need for domain-specific fine-tuning. This work provides data-driven guidance for deploying open-source models in ESG reporting.