Japanese financial exams power new updatable language model test set

JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams

Artificial Intelligence

Summary

Language models like ChatGPT need special tests to see how well they understand finance in different countries. Existing tests often don’t fit specific countries or get outdated when their questions leak online. The authors made a new test set called JFinBench from recent Japanese finance exams that keeps updating. They created a process where a computer program helps format exam questions but a person checks and corrects everything, making updates easier. Testing several language models showed some perform very well, especially when they can also search the web or use a terminal, but the model’s own knowledge still matters.

What this means in practice

  • For financial ai developers: Use the JFinBench dataset and pipeline to test and improve language models’ Japanese finance knowledge with up-to-date exam questions.
  • For ai tool integrators: Incorporate tools like web search and terminal access to models when handling Japanese financial queries for better accuracy.

Authors

Issa Sugiura, Taro Makino, Takashi Ishida

Abstract

Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.