Stateless service gives repeatable text changes preserving original meaning
perturber: A Stateless Service for Semantics-Preserving Text Perturbations
Computation and Language
Summary
Evaluations of language AI often miss how small changes in sentences can confuse these systems even when the meaning stays the same. The authors created perturber, a tool and service that makes one small, meaning-preserving change to a piece of text each time it runs, ensuring consistent results no matter where or when it's used. This helps others test how robust language models really are by applying known, measured text changes one at a time. They also provide a live demo and library to use these changes easily for evaluation.
What this means in practice
- •For natural language processing engineers: Apply single, measured text changes reproducibly to evaluate and improve language model robustness consistently.
- •For automated content moderation teams: Test moderation systems against slight text modifications that keep original meaning to reduce false negatives in detection.
Authors
Khoi-Nguyen Tran
Abstract
Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer. We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families. It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction. We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model's robustness, measuring meaning-preservation per perturbation rather than assuming it. The demonstration adds a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation.