LLM robustness tested using noisy and altered input tasks
LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances
Computation and LanguageArtificial Intelligence
Summary
Knowing how well large language models (LLMs) handle mistakes or changes in text is important to trust their answers. The authors created a set of tasks that mix up words, repeat characters, and change question options to see how these models react. They tested many models of different sizes and types to find out which ones handle errors better or worse. Their work helps show areas where models need to improve so they can give more reliable answers even with messy input.
What this means in practice
- •For machine learning engineers: Improve robustness testing workflows by integrating a diverse dataset of noisy and transformed tasks for evaluating language models under various input errors.
- •For quality assurance teams: Develop better evaluation protocols to detect when language models fail due to input perturbations, improving model reliability in deployed systems.
Authors
Vamsi Krishna Kodavali, Rituraj Singh
Abstract
Robustness evaluation of large language models (LLMs) remains a critical challenge, particularly in assessing their sensitivity to perturbations in input data. In this work, we systematically evaluate LLM robustness across multiple dimensions, including word error rate, character repetition and duplication, modifications in choices, and variability in instruction following. To facilitate this evaluation, we construct a synthetic and augmented dataset encompassing a diverse set of LLM benchmarks, specifically targeting multiple-choice question (MCQ) datasets and instruction-following tasks. We conduct extensive experiments on LLMs of varying scales-small, medium, and large-as well as across base and instruction-tuned variants. Our analysis quantifies the variability in model responses under perturbed conditions and highlights discrepancies relative to baseline models. The findings provide insights into the stability of LLMs across different evaluation scenarios contributing to the development of more robust and reliable language models as well as robust evaluation methodologies.