French spelling changes reduce language model accuracy and reliability
Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants
Computation and Language
Summary
French speakers often type words without accents or make small keyboard mistakes, but it wasn’t clear how these changes affect smart language programs. The authors created a test with many French sentence variations, from mild to strong changes. They found that even small edits lower the models’ accuracy and how reliable their confidence is, especially with typos. Larger models lose less accuracy, but their confidence scores don’t always get better. Fixing inputs and adjusting confidence together helps improve trustworthiness and accuracy.
What this means in practice
- •For natural language processing teams: Improve French language model performance by combining input correction with confidence calibration for noisy user inputs.
- •For user interface developers: Design text input systems that anticipate common French typing errors to maintain AI reliability despite input variations.
Authors
Jiaming Yu, Xiaotian Xie, Jian Ma, Yingjie Li
Abstract
French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets (12,600 input instances). Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood (NLL) increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.