BenchCompass reveals challenges for language models in payments tasks
BENCHCOMPASS: From Scores to Signals for Training and Harness Decisions in Payment-Domain LLMs
Artificial Intelligence
Summary
Payment systems are complicated and change often, making it hard for AI language models to help with payment decisions. The authors created BenchCompass, a test set that checks if models understand payment rules, use evidence well, and handle tricky situations. They found that current models still struggle with missing knowledge, reasoning, and rejecting wrong payment steps. Their work provides a way to measure AI progress in this area.
What this means in practice
- •For financial software teams: Evaluate language models’ ability to follow complex payment rules and handle varied payment scenarios for improving automated payment decision tools.
- •For ai model developers: Use BenchCompass to diagnose specific failure modes in payment-related language models and guide model improvements and training.
- •For customer support platforms: Incorporate tests from BenchCompass to ensure chatbot assistants handle payments questions accurately and reject invalid payment workflows.$Commercial implications: Enables development of commercial support bots compliant with payment domain rules and robustness standards.
Authors
Sijie Dong, Wei Ren, Xuanwei Hu, Jiawei Luo, Zifan Wang, Xiaoyun Feng, Hui Cai, Lyuxin Xue, Peng Lu, Jianshe Li, Xin Zhang, Wei Wu
Abstract
Payment operations are a critical financial infrastructure, but the value of large language models in this domain remains unclear because payment rules change quickly, evidence is fragmented, and decisions depend on transaction state, participant role, region, and payment rail. Existing benchmarks do not isolate whether failures come from missing payment-rule knowledge, poor use of supplied evidence, or brittleness under imperfect harness inputs. We introduce BENCHCOMPASS, a payment-domain benchmark whose construction pipeline builds scenario-grounded tasks from typed evidence packs, applies LLM-based quality checks, creates task-input attack variants, and reserves final item admission for domain experts. The release contains an expert-reviewed Pro benchmark covering payment knowledge, context-grounded scenario reasoning, and Attacked Open robustness, plus a lower-assurance Normal pool for inspection and future curation. Across 16 model variants, BENCHCOMPASS shows qualitatively different failure modes: missing parametric payment knowledge, incomplete reasoning over supplied rules, and failure to reject plausible but invalid workflows. The benchmark remains unsaturated: the best frontier model reaches 89.6% on Open Context-Grounded Reasoning and 81.7% under attacked inputs, while a representative 32B open-weight model reaches 69.8% and 42.6%. Benchmark data and code are available at https://github.com/ant-intl/BenchCompass.