Evaluating safety and reliability of Indian banking chatbots
IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Artificial Intelligence
Summary
Chatbots that help with banking need to use your specific account information correctly to answer questions and perform actions. The authors created IndicBankBench, a set of 799 testing cases that check not only the final answers but also whether the chatbot safely uses tools, avoids repeating questions, and picks the right account. They tested 11 language models and found that getting the right result consistently is harder than it seems. This tool helps identify specific weak points in banking assistants so they can be improved.
What this means in practice
- •For banking chatbot developers: Use IndicBankBench to detect specific safety and reliability issues in Indian retail banking assistants before deployment.
- •For financial compliance teams: Assess the dependability of automated banking responses in compliance reviews by evaluating tool use and context handling.
Authors
Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga
Abstract
Banking assistants must use account-specific information to answer requests and, in many cases, take actions through tools. Evaluating only the final response misses important errors. An assistant may ask for information it already has, rely on stale context, select the wrong account, or write an invalid value after stating the correct one. We introduce IndicBankBench, a 799-case benchmark for Indian retail banking spanning five operational domains, a capability/refusal domain, and twenty primary axes. Cases are evaluated at four stages: safety, action and tool use, response adequacy, and advisory quality. Tool use and most safety checks are deterministic. A narrow resolver handles only ambiguous confirmation-before-write cases, while a separate LLM judge evaluates semantic response adequacy. We run every case three times and report strict pass^3, which requires success on all trials. Across the eleven evaluated models, strict reliability ranges from 43.7% to 58.2%, whereas at-least-once success ranges from 60% to 74%. This gap shows that at-least-once success can overstate dependable banking behavior. The case-level diagnostics also distinguish systems that ask unnecessary questions from those that act but fail to reconcile customer context or fully resolve the request. We release the cases, mock environment, and evaluation harness.