Japanese benchmark improves tool-using AI agents for customer support
J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction
Computation and Language
Summary
Evaluating AI assistants that use external tools for customer support is important, but current tests mostly focus on English, making it hard to check how well they perform in Japanese. The authors created J-tau, a Japanese version of a popular English benchmark, carefully translating and adapting it to fit Japanese contexts like addresses and currencies. They also found and fixed problems in the original test that made some AI models look better than they really are. Their version works more reliably for Japanese and helps better evaluate AI tools in that language.
What this means in practice
- •For customer support teams: Evaluate Japanese-language AI agents that use external tools for interactive customer support efficiently and reliably.
- •For multilingual ai developers: Test and improve Japanese tool-using AI agents with a benchmark that reflects local language and cultural details accurately.
Authors
Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa, Ryota Hirobuchi
Abstract
We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While tau-bench is widely used to evaluate LLM agents, it is English-centric and cannot evaluate agents in other languages. Existing multilingual extensions offer no public Japanese version and rely primarily on machine translation, leaving locale-specific entities such as addresses and currencies inconsistent with the target region. To enable reliable evaluation in Japanese, we carefully localize tau-bench through machine translation, entity adaptation, and human review. During this process, we also uncover and fix two flaws in the original tau-bench: invalid tasks and a score aggregation error that inflates certain models' results. Experiments show that J-tau yields fewer evaluation errors than a naively machine-translated counterpart and that our fixes resolve both flaws.