AfriGuard improves safety in African language AI with better refusal detection
AfriGuard: Safety and Robust Refusal For African Languages
Computation and Language
Summary
Chatbots and AI models need to be careful responding safe and polite, especially in many African languages where meanings vary. The authors created AfriGuard, a dataset to help AI understand harmful content and refuse to answer in 10 African languages plus English. They found many existing models either misclassify unsafe content or avoid proper refusal, giving useless answers instead. Using AfriGuard, they built smaller, smarter models that match much larger ones in safety and refusal accuracy. This helps AI behave better in diverse languages and cultures.
What this means in practice
- •For language technology developers: Create safer multilingual chatbots that refuse harmful requests effectively across African languages and English.
- •For content moderation teams: Improve automatic detection and refusal of unsafe or harmful user inputs in African languages to enhance platform safety.
Authors
Israel Abebe Azime, Jesujoba O. Alabi, Iffat Maab, Alexander D. Cai, Tadesse Destaw Belay, Atnafu Lambebo Tonja, Jonas Kgomo, Folasade Alabi, Salomey Osei, Saminu Muhammad Aliyu, Nkechinyere Faith Aguobi, Bontu Fufa Balcha, Blessing Sibanda, Davis David, Mouhamadane Mboup, Daud Abolade, Neo Putini, Louis Powell, Kiant'e Brantley, Philipp Slusallek, Dietrich Klakow, Julia Kreutzer
Abstract
Large language models (LLMs) designed to interact with users are typically trained to respond appropriately to harmful or toxic content. However, ensuring appropriate responses becomes more challenging in diverse linguistic and cultural contexts, where safety risks may be expressed or interpreted differently. Building safe AI systems therefore requires detecting safety risks, understanding their nature, and responding appropriately, while accounting for linguistic and cultural variations that add complexity to each stage. In this work, we introduce AfriGuard, a culturally grounded multilingual dataset for studying harmful intent, safety-category understanding, and refusal behavior across 10 African languages and English. Our findings reveal that guardrail models underperform both open- and closed-source LLMs on safety-intent and safety-category classification, while models show limited understanding of appropriate safety behavior, scoring below 60% F1. We further find that models often fail to respond appropriately to unsafe prompts not because they explicitly refuse, but because they produce linguistically inadequate responses that result in incidental non-assistance rather than proper refusals. Finally, using AfriGuard, we develop efficient guardrail models that achieve performance competitive with models up to 30x larger, introduce refusal detectors that improve average performance by 43.9% over existing detectors, and demonstrate that supervised fine-tuning can substantially improve overall safety performance.