Text safety API misses some harmful messages and rarely flags harmless ones
SITO: Sifting a Commercial Text-Safety API for Missed Harm and False Flags in English, Ukrainian and Surzhyk
Computation and Language
Summary
Many online platforms use commercial tools to catch harmful messages, but it's unclear how well these tools work, especially in languages other than English. The authors studied one such tool called Tuteliq Detect and tested it with thousands of made-up messages in English, Ukrainian, and a mixed language. They found that the tool catches most harmful messages but sometimes misses reports from third parties about ongoing harm. It also rarely mistakenly flags harmless messages. The findings suggest platforms should test these tools carefully with their own users' messages.
What this means in practice
- •For platform content moderators: Measure false positives and missed harm using synthetic messages similar to your users’ languages and topics to better understand your text-safety API’s performance.
- •For multilingual chatbot developers: Test chat moderation tools on English, Ukrainian, and mixed-language data to ensure consistent detection of harmful content across languages.
Authors
Andrii Tuzhykov
Abstract
Platforms that host young users screen messages with commercial text-safety APIs, but independent evidence on how often such systems miss harm or flag harmless text is scarce, especially outside English, and the documentation of the one studied here leaves its false-positive rate unquantified. We evaluated one such API, Tuteliq Detect as deployed under the release identifier tuteliq-detect-v5.20261011 (not the separately published v6 model), on six harm categories: authorised push-payment fraud, money-mule recruitment, social engineering, romance scams, gambling harm and bullying. We built SITO (Synthetic Independent Text-safety benchmark with Overflagging tests; sito is Ukrainian for 'sieve'), 2,332 synthetic cases in English, Ukrainian and mixed Ukrainian, with each harmful case matched by hard harmless ones: same-topic messages, minimal-edit contrasts, warnings and retrospective accounts. On English text the system flagged 84.9-94.1% of harmful cases and 0-4.3% of harmless ones per category. It missed third-party reports of ongoing harm far more often than perpetrators' messages (63.6% against 95.1% recall, all languages), when sent without the settings the API offers for reports of harm. Ukrainian and mixed-language input was detected at least as often as English, recall was lower on text from a held-out generator family, so headline recall may be optimistic, and 3.2% of cases received a different decision in repeated runs. A platform should measure false positives on its own traffic and check how bystander reports are handled. The datasets, generation prompts and code are released.