Language specific thresholds improve text classification accuracy

One Threshold Does Not Fit All Languages: Language-Conditional Deferral for Reliable and Efficient Low-Resource Text Classification

Computation and LanguageMachine Learning

Summary

Many language models are used to classify text in multiple low-resource languages but often fail to meet accuracy promises equally across all languages. The authors found that using one set confidence threshold for many languages underperforms some, especially those with less data. By setting thresholds separately for each language, the system becomes more reliable, though it means sending many more texts for human review in some languages. This approach requires only a small number of labeled examples per language and does not need retraining.

What this means in practice

  • For multilingual nlp engineers: Calibrate text classifiers separately for each language to improve reliability and better manage human review workloads.
  • For content moderation teams: Use language-conditional deferral to allocate automated labeling confidently while efficiently prioritizing review effort by language.

Authors

Bhanu Prakash Vangala, Vangala Navya

Abstract

In the Global South, the lower-income countries of Africa, Asia, and Latin America where most of the world's languages are spoken, a deployed text classifier usually runs on ordinary CPUs, serves many languages with a single model, has few labeled examples in any of them, and relies on people to catch its mistakes. Such a system is only useful if it can promise how often it will be wrong: at most a fixed fraction of the labels it assigns on its own may be incorrect, and everything else must go to a person. Split conformal prediction delivers this promise through a single confidence threshold, normally estimated on validation data pooled across languages. We ask whether the promise reaches every language, and it does not. On MasakhaNEWS (16 African languages) and AfriSenti (12 languages plus two never seen in training), a pooled threshold meets the 90% target on average but covers Somali at 77.5%, Tigrinya at 83.7%, and the two unseen languages at 77.5% and 81.2%. Estimating one threshold per language brings every language to between 89.1% and 91.0% without retraining, and it shows how unequal the cost of the promise is: keeping it means sending 43% of Somali news and over 80% of Amharic and Xitsonga tweets to a person, against under 8% of Nigerian Pidgin news. One or two hundred labels per language are enough and the models train in minutes on one CPU core, so the fix is affordable: calibrate, report, and budget human review one language at a time.