Large language models improve hard named entity recognition in Nepali tweets
Using an LLM to Tackle Difficult Cases in NER for Nepali Tweets
Computation and Language
Summary
Named entity recognition (NER) tries to find names of people, places, and things in text. It is hard to do well when the text is messy social media posts and the language is less commonly studied, like Nepali. The authors split tweet examples into easy and hard groups using a special kind of language model. They then tested a large language model (LLM) on the hard examples and found it did better than regular models on those cases. By combining both models, they achieved better overall performance.
What this means in practice
- •For social media analysis teams: Improve entity detection in difficult Nepali social media posts by integrating LLMs targeting hard cases.
- •For multilingual nlp engineers: Use example selection by semantic similarity and increasing prompt examples to boost LLM performance on hard NER tasks in low-resource languages.
Tested on one dataset.
Authors
Claire Wang
Abstract
Named entity recognition (NER) is challenging for social media data in a low-resource language. LLMs often lag behind pretrained language models (PLMs) in NER performance, but they show promise in handling cases that PLMs struggle with. Using a dataset of Nepali tweets, we split the test set into "easy" and "hard" cases using token probabilities from a fine-tuned pretrained language model (PLM). We then evaluate an LLM on the hard test cases using 16 different prompt configurations. We find that selecting examples based on semantic similarity and increasing the number of examples improves F1 score by almost 10 points from the baseline configuration. Although the LLM underperforms the PLM on the full and easy test sets, it surpasses the PLM on the hard test set, resulting in an improved score when the two models are combined.