Improving bilingual banking search accuracy with evidence gating

Evidence-Gated Bilingual RAG for Reliable Consumer Banking Search

Computation and Language

Summary

People searching banking websites often use everyday language that doesn’t match official website terms, making it hard to find the right info. This problem is even harder when users use two languages like Arabic and English. The authors developed a new system that carefully picks and checks information from banking websites to better understand and answer user requests in both languages. Their tests show big improvements in matching searches to the right banking pages and in generating trustworthy answers, though more testing is needed to confirm results fully.

What this means in practice

Authors

Ahmed Mohamed Mahmoud Rayyan, Rana Mahmoud Mohammed, Aya Abdallah Abdelmonem, Mohamed Khaled Goda Othman, Rahatara Ferdousi

Abstract

Consumers often describe what they need rather than use the terminology found on a banking website. For example, a natural-language request ("a card for travel") may not share the keywords of the relevant product page, and the problem is harder in bilingual services, where the same intent may be expressed in Arabic or English. This paper presents an evidence-gated bilingual retrieval-augmented generation (RAG) framework for consumer banking search that operates entirely on local infrastructure. The framework combines structured web-content refinement, hybrid dense-lexical retrieval, intent-aware evidence selection, cross-encoder reranking, and a three-outcome Answer / Related Mismatch / No Answer decision mechanism. On 210 URL-labeled Arabic and English queries, the proposed approach improves Top-1 hit rate from 50.48% to 79.05% and MRR@5 from 0.571 to 0.797 over a hybrid baseline. A separate 26-case end-to-end evaluation achieved 69.2% strict success, with 72.7% of generated answers classified as supported by an automated faithfulness judge. These results are preliminary: the retrieval queries are paraphrases of a small reference set, and the end-to-end test is small and uses an automated rather than human judge, so they do not establish human-verified answer correctness or generalizable performance.