Label wording changes improve accuracy for unfamiliar words in typed-decision models
Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Some computer programs decide what to do by choosing from a list of options described by words. The authors found that changing how these option labels are written can help the program better understand unfamiliar or borrowed words, like jargon or slang. However, these word changes do not help when the options are abbreviations or codes, which the program struggles with. Their tests showed that the improved way of writing labels helped accuracy, especially when using regular words, but sometimes caused more confident mistakes with many options.
What this means in practice
- •For natural language processing engineers: Improve classification accuracy for models handling unfamiliar or borrowed words by augmenting label wording during inference.
- •For customer service chatbot developers: Enhance chatbot correctness on options with descriptive labels by varying their wording, especially when options contain slang or jargon.
Authors
Vladimir Beskorovainyi
Abstract
Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies, produced by three separate processes and checked for overlap with the training data, with five candidates per question and, for eight of them, also with eighteen. On held-out vocabularies made of ordinary or borrowed words (English-derived jargon, office slang, department names), the augmented release is more accurate in every setting, by up to 0.24. Eleven of these fourteen gains survive a Holm correction (slang with eighteen candidates and descriptive sentences with five do not), and for jargon on all labels the label-level interval excludes zero in both settings. On abbreviations and opaque codes it gives no usable gain: on abbreviations it is not better, and worse on the clean slice, and on opaque codes both releases stay at or below chance once labels related to training options are removed. In most cells where accuracy rises, the risk-coverage area falls, but with eighteen candidates confident errors rise on jargon and slang. On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points. We also report four evaluation hazards from comparing the two releases.