Models improve answering European financial rules but cite wrongly
FinRegQA-EU: Corruption-Based Preference Data for Grounded EU Financial Regulatory Question Answering
Computational Engineering, Finance, and Science
Summary
Large language models have trouble giving accurate answers about European financial regulations. The authors made a special test set using official questions from European regulators to check these models. They found that some training methods make models give longer answers with more citations, but many citations are made up. Other methods produce shorter answers with more accurate citations. This shows that simply counting citations can be misleading when checking if answers are trustworthy.
What this means in practice
- •For financial compliance teams: Evaluate and improve AI tools to answer European financial regulatory questions with better citation accuracy and trustworthiness.
- •For legal tech developers: Develop AI systems that provide well-grounded answers on EU financial laws using the new dataset and evaluation methods to reduce fabricated citations.$Commercial implications: Enables creation of compliant legal QA products that must verify citations for accurate advice to financial institutions.
Authors
Aulia Kharis Rakhmasari, Fan Yu, Alexander Hoyle, Elliot Ash
Abstract
Large Language Models (LLMs) struggle with region-specific factual knowledge, particularly in financial regulation. While benchmarks such as CFinBench, provide broad coverage of financial knowledge in other regions, no comparable resource exists for European financial regulation. We close this gap by proposing an end-to-end pipeline for evaluating and improving LLMs on European financial regulatory question answering. Our dataset is grounded in the official Q&A corpora of the European Banking Authority (EBA) and the European Securities and Markets Authority (ESMA). We evaluate candidate answers with a pointwise LLM-as-a-Judge protocol using three judges from distinct model families, retain only unanimously judged pairs, and sharpen the rejected side through a taxonomy of regulatory failure modes : law swaps, article swaps, hallucinated citations, and hallucinated text. By fine-tuning on the resulting preference pairs exposes a divergence at the core of our findings: supervised fine-tuning attains the highest judge score of any method yet got the lowest rule-based Citation F1. SFT is producing longer answers with three times as many citations, most of them unsupported. DPO and GRPO offer the better trade-off with more concise answers and the highest citation F1 among fine-tuned models. Standard LLM-as-a-Judge evaluation rewards citation density as evidence of grounding and cannot detect when those citations are fabricated. Thus, we detect a blind spot that matters wherever answers must be verifiable. We release the benchmark and evaluation stack at https://github.com/auliakharis/FinRegQA-EU.