Chinese language debate data offers new test for AI argument skills
Chinese Competitive Debating Dataset and Benchmark
Computation and LanguageArtificial Intelligence
Summary
Judging debates is hard because it requires understanding how arguments change during the discussion. The authors collected detailed Chinese debate records judged by professionals in real competitions to create a new dataset. They used this data to test AI models on their ability to predict winners, scores, and top debaters. The best AI models were correct about two-thirds of the time at predicting winners but struggled more with the other tasks. This dataset helps study how well AI can understand complex, interactive arguments in Chinese debates.
What this means in practice
- •For ai developers: Test and improve language models on their ability to understand interactive Chinese debates using detailed real competition data.
- •For debate platform operators: Incorporate AI support tools trained on this dataset to assist professional judging and feedback in Chinese competitive debates.
Authors
Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Zhirui Zeng, Pengqian Han, Yi Zhou, Yuting Wang, Jiamou Liu
Abstract
Debate adjudication requires tracking how arguments develop through interaction, yet existing datasets rarely combine fine-grained debate transcripts with professional judgments collected during real competitions under a shared rubric. We introduce a dataset and benchmark for evaluating large language models' understanding of competitive Chinese-language debate at the match, stage, and speaker levels. We organized 182 matches and recruited 120 professional judges, with each match independently adjudicated by three judges using a predefined rubric. After excluding matches with incomplete records, the dataset contains 148 matches, 2,698 stages, and 20,542 exchange units, with manually verified transcripts and segmentation. It preserves original stage scores, match votes, best-debater ballots, and adjudication rationales. We define three tasks: winner-tendency prediction, stage-score prediction, and best-debater prediction. Zero-shot evaluation of multiple large language models yields a highest winner-prediction accuracy of 66.2%, a highest Pearson correlation of 0.250 between model stage scores and mean human ratings, and a highest best-debater prediction accuracy of 56.8%. The dataset and benchmark provide a testbed for studying large language models' understanding of interactive argumentation and their agreement with professional judges.