Stateless service gives repeatable text changes preserving original meaning

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Computation and Language

Summary

Evaluations of language AI often miss how small changes in sentences can confuse these systems even when the meaning stays the same. The authors created perturber, a tool and service that makes one small, meaning-preserving change to a piece of text each time it runs, ensuring consistent results no matter where or when it's used. This helps others test how robust language models really are by applying known, measured text changes one at a time. They also provide a live demo and library to use these changes easily for evaluation.

What this means in practice

Authors

Khoi-Nguyen Tran

Abstract

Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer. We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families. It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction. We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model's robustness, measuring meaning-preservation per perturbation rather than assuming it. The demonstration adds a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation.