Language models tested for errors and consistency under different input changes
LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances
Computation and LanguageArtificial Intelligence
Summary
It is hard to know how well language models handle mistakes or changes in what they are asked. The authors created special test sets that mess with the input in ways like repeating letters or changing answers to multiple-choice questions. They checked how small, medium, and large language models, including ones trained to follow instructions, reacted to these changes. Their work shows which models stay steady and which get confused, giving clues about how to make language models more trustworthy. This also helps improve ways to check how strong these models really are.
large language modelsrobustnessperturbationsmultiple-choice questionsinstruction tuningsynthetic dataseterror ratemodel evaluation
Authors
Vamsi Krishna Kodavali, Rituraj Singh
Abstract
Robustness evaluation of large language models (LLMs) remains a critical challenge, particularly in assessing their sensitivity to perturbations in input data. In this work, we systematically evaluate LLM robustness across multiple dimensions, including word error rate, character repetition and duplication, modifications in choices, and variability in instruction following. To facilitate this evaluation, we construct a synthetic and augmented dataset encompassing a diverse set of LLM benchmarks, specifically targeting multiple-choice question (MCQ) datasets and instruction-following tasks. We conduct extensive experiments on LLMs of varying scales-small, medium, and large-as well as across base and instruction-tuned variants. Our analysis quantifies the variability in model responses under perturbed conditions and highlights discrepancies relative to baseline models. The findings provide insights into the stability of LLMs across different evaluation scenarios contributing to the development of more robust and reliable language models as well as robust evaluation methodologies.