Ai text detectors often misidentify kazakh writing before chatgpt era
Unchecked or Falsely Accused? Evaluating AI-Generated Text Detectors on Kazakh
Computation and Language
Summary
Detecting AI-written text can be tricky especially in languages like Kazakh. The authors studied detectors trained in English and found they often wrongly flag genuine human Kazakh writing from before AI models like ChatGPT existed. They created and tested new ways to improve detection accuracy, reducing false alarms while catching more AI-generated Kazakh text. Their improved detector works better with different scripts and text lengths, and they shared their data and tool openly.
What this means in practice
- •For content moderation teams: Use improved AI text detectors customized for Kazakh to reduce false positives and better identify AI-generated content in user submissions.
- •For online education platforms: Apply calibrated detection tools to check for AI-generated essays in Kazakh, minimizing incorrect accusations of human authorship.
Authors
Mansur Ozaman
Abstract
AI-text detectors are already applied to Kazakh writing, yet no study reports how often publicly available detectors flag Kazakh text whose human authorship is dated before ChatGPT. We present AINA-KK: 3,811 human passages from encyclopedic, news, web and educational pages that provably existed before ChatGPT, and 5,597 machine and gray-zone passages from 9 generators, including the Kazakh-built KazLLM and Sherkala, with script and length stress tests. Five English-trained detectors from the Hugging Face Hub flag up to 81.4% of human passages and 79.6% of published student essays; recalibrated, they catch at most 1.3% of machine text. Calibrated zero-shot detectors catch at most 50.3% of machine text from in-distribution generators. Fine-tuned encoders rank text almost perfectly, but mDeBERTa at its default threshold flags 24.4% of 50-word human excerpts, partly because machine outputs are shorter than human text; calibrated, it catches only 47.1% of KazLLM and 62.7% of Sherkala passages. Without script handling, machine-transliterated Latin-2021 text defeats every detector. Fixes that need no new data (calibration, script canonicalization, script-augmented and length-matched training, and an OR with a zero-shot scorer) form AINA-KK-Det. On test it flags 0.9% of human passages (95% CI 0.4-2.7; 1.5% on development data) and catches 85.2% of machine text from known generators, 92.9% from two unseen Gemini-family Gemma-4 models and 61.3% from the Kazakh-built ones; its configuration was chosen with test results in view. We release the data, code and detector at https://github.com/ozxmn/aina-kk.