Chinese human AI writing collaboration challenges text detection reliability
C-HAT-Bench: Benchmarking Chinese AI-Text Detection Beyond Fully Generated Text
Computation and Language
Summary
Detecting whether a piece of writing was made by AI is usually done by comparing fully human-written text with fully AI-generated text. But when humans and AI work together on the same text, it becomes much harder to detect AI involvement. The authors studied this problem in Chinese by creating a large benchmark with many examples of human-AI mixed writing from different domains and AI models. They found that current AI text detectors perform much worse on these mixed texts and that success in detecting one type of collaboration doesn’t predict success in others.
What this means in practice
- •For content moderation teams: Use the benchmark to evaluate and improve AI text detection tools for more complex, mixed human-AI written Chinese content.
- •For language technology developers: Train and test AI text detectors with diverse human-AI collaboration styles in Chinese to build more robust detection systems.
Authors
Qing Yang, Zixiang Luo, Zhenyu Mao, Zezheng Wu, Xinghe Cheng, Haibo Chen, Qinggang Zhang, Jiapu Wang, Jingwei Zhang
Abstract
Large Language Models (LLMs) increasingly participate in writing by modifying or extending human drafts, causing machine involvement to vary in both form and extent. Yet most Machine-Generated Text (MGT) detectors are evaluated only on fully human-written versus fully AI-generated text. Because human--AI collaboration can weaken or redistribute cues associated with machine generation, strong performance under this binary setting may overstate detector reliability. This mismatch remains underexplored in Chinese: detection cues are shaped by tokenization and language-specific text distributions, yet controlled resources spanning production settings, domains, and generators remain limited. To fill this gap, we present a Chinese Human-AI Collaborative Text Detection Benchmark (C-HAT-Bench), a unified benchmark that links $5,000$ human-written source texts from five domains to more than $240,000$ variants produced using six generative models under Prefix-Conditioned Continuation as a reference setting and three collaborative production modes. We evaluate $21$ detectors through four protocols spanning zero-shot and pretrained supervised document-level detection, boundary localization, and cross-condition generalization. Relative to Prefix-Conditioned Continuation, mean AUROC across document-level detectors is $12.0\%$ lower on the collaborative production modes, with the largest detector-specific relative decrease reaching $44.4\%$. Transfer across collaborative production modes is also asymmetric, indicating that performance in a given production setting is not a reliable predictor of performance in other production settings.