TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice
2026-04-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created TaxPraBen, a new benchmark to test how well large language models (LLMs) handle real-world Chinese tax tasks, going beyond simple language tests. It includes 10 common tax tasks and 3 real scenarios like tax risk prevention, using data from 14 sources. They tested 19 LLMs and found big differences: large closed-source models did best, Chinese models often outperformed multilingual ones, but fine-tuning on tax data helped only a little. TaxPraBen aims to better measure practical tax knowledge in LLMs and can be adapted for other fields.
Large Language ModelsChinese TaxationBenchmarkTax Risk PreventionTax Inspection AnalysisTax Strategy PlanningBloom's TaxonomyFine-tuningStructured EvaluationNatural Language Processing
Authors
Gang Hu, Yating Chen, Haiyan Ding, Wang Gao, Jiajia Huang, Min Peng, Qianqian Xie, Kun Yu
Abstract
While Large Language Models (LLMs) excel in various general domains, they exhibit notable gaps in the highly specialized, knowledge-intensive, and legally regulated Chinese tax domain. Consequently, while tax-related benchmarks are gaining attention, many focus on isolated NLP tasks, neglecting real-world practical capabilities. To address this issue, we introduce TaxPraBen, the first dedicated benchmark for Chinese taxation practice. It combines 10 traditional application tasks, along with 3 pioneering real-world scenarios: tax risk prevention, tax inspection analysis, and tax strategy planning, sourced from 14 datasets totaling 7.3K instances. TaxPraBen features a scalable structured evaluation paradigm designed through process of "structured parsing-field alignment extraction-numerical and textual matching", enabling end-to-end tax practice assessment while being extensible to other domains. We evaluate 19 LLMs based on Bloom's taxonomy. The results indicate significant performance disparities: all closed-source large-parameter LLMs excel, and Chinese LLMs like Qwen2.5 generally exceed multilingual LLMs, while the YaYi2 LLM, fine-tuned with some tax data, shows only limited improvement. TaxPraBen serves as a vital resource for advancing evaluations of LLMs in practical applications.