Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models (LLMs) are used to create labeled data for training smaller models that find named entities in text. They found current methods ignore mistakes made by LLMs in the labels, which hurts performance. Standard techniques to handle noisy labels don't work well here because errors in named entity recognition come in different types, like missing entities or wrong entity types. The authors propose a new way to handle different error types separately during training, which improves model accuracy without extra training costs. Their approach boosts performance noticeably on noisy datasets like Wikigold.
large language modelsnamed entity recognitionannotation noisefine-tuningnoise-robust losserror reweightingsequence labelingF1 scoresynthetic datasetWikigold
Authors
Elena Merdjanovska, Jonas Golde, Alan Akbik
Abstract
Large language models are increasingly used to annotate datasets for training smaller, task-specialized models such as named entity recognition. While this method yields effective models, it assumes that the synthetic dataset is correctly annotated. In this work, we find that (i) current fine-tuning processes simply ignore LLM-introduced annotation noise, resulting in degraded performance and (ii) existing noise-robust losses are not transferable to sequence labeling because annotation noise in named entity recognition is heterogeneous: for example, missing mentions and type errors affect the training signal in different ways. Treating all noisy tokens equally in noise-robust losses and applying a single reweighing criterion for all may therefore remove useful supervision or reinforce incorrect labels. To address this limitation, we propose error-type-aware loss reweighting for NER, which introduces separate reweighing rules for different types of potentially erroneous tokens. Our approach is simple and efficient, does not require additional training resources, and improves F1 by 0.8 - 2.0 percentage points on dataset-level average for noise levels between 15% and 40%, with a maximum improvement of 4.6 percentage points with 24.1% noise on Wikigold.