DeBERTa-ConPara improves detection of AI-generated text with attack defenses
DeBERTa-ConPara: Attack-Aware and Deployment-Realistic Detection of AI-Generated Text
Computation and LanguageCryptography and Security
Summary
Detecting AI-written text is hard because people try to hide it using tricks, and the tools often struggle when facing new types of AI or different topics. The authors developed DeBERTa-ConPara, which uses smart text processing and a powerful AI model trained on various datasets to better spot AI-generated content even when hidden by tricky attacks. They found that cleaning text during detection, but not during training, works best to catch hidden tricks like replacing letters with lookalikes. However, their added techniques like paraphrasing and feature blending didn’t improve detection, showing some common ideas don’t always help.
What this means in practice
- •For content moderation teams: Deploy an improved AI text detector that better catches disguised AI-generated content under realistic attack scenarios.
- •For social media platform operators: Use the detection approach to enhance automated filters against misleading posts created or manipulated by AI with subtle text changes.
Authors
Mohamed Mady, Yupei Li, Johannes Reschke, Björn W. Schuller
Abstract
Robust detection of AI-generated text under deployment conditions is challenging: distribution shifts across domains and generators, adversarial perturbations of the input surface, and the absence of target-domain labels for threshold calibration all degrade detectors that perform well in-domain. We present DeBERTa-ConPara, a deployment-oriented detector combining attack-aware Unicode preprocessing with a contextual transformer encoder trained over HC3 Plus, M4, MAGE and RAID. Our central finding is that preprocessing acts in opposite directions depending on where it is applied: normalising the training corpus deduplicates it, collapsing 35.4% of RAID rows into copies of their clean siblings and deleting the adversarial supervision, whereas normalising at inference is an effective defence. A factorial varying the two placements independently identifies raw training with normalised inference as the best configuration, reaching 99.61% AUROC, 99.01% TPR@5% FPR and 96.57% TPR@1% FPR on the official RAID hidden test, alongside 93.14% average balanced accuracy across HC3 Plus and MAGE under a fixed threshold. The gain is confined to two of twelve attack classes: homoglyph and zero-width-space insertion rise from 11.05% and 1.12% to 96.98%. The same signature reproduces in a zero-shot detector of different architecture, showing the effect belongs to the attacks rather than to our model. We additionally report two negative results: semantic-invariance augmentation through paraphrasing and supervised contrastive learning (ConPara) does not improve the best configuration, and the handcrafted feature-fusion branch is inert in distribution and harmful outside it.