Padārtha: Ontology-Grounded Fine-Grained NER Benchmark for Classical Sanskrit
Computation and Language
Summary
The authors highlight that standard labeling systems used to identify named entities aren't well suited for ancient texts like the Sanskrit epic Mahābhārata. They created a new labeling system called Padārtha based on a classical Indian ontology to better capture fine details of entities in this text. Using expert annotators, they labeled over 100,000 entity mentions and tested different machine learning models to recognize these entities. They found that while newer generative models work about as well as traditional ones, all models struggle with very detailed categories and recognizing new or ambiguous entities. This problem isn't just about having enough data but also about how models handle unfamiliar or unclear cases.
Authors
Sujoy Sarkar, Pretam Ray, Paramhans Shah, Manoj Balaji Jagadeeshan, Akash Gairola, Arjuna S R, Pawan Goyal
Abstract
Annotation schemas are not neutral. When applied to classical literature, tag sets developed for modern journalistic texts impose source-culture definitions on texts they were never designed to describe. We instead ground a schema in the tradition of the text itself introducing \textit{Padārtha}, the first ontology-grounded fine-grained Named Entity Recognition (NER) benchmark for Sanskrit, built on the \textit{Mahābhārata} epic. Our tag set derives from \textit{Nyāya-Vaiśesika}, a classical Indian ontological system, yielding 18 fine-grained categories organized under 10 ontological nodes and mapped onto five standard coarse tags, ensuring interoperability with existing benchmarks. Expert annotators label over 12.6K entries from a scholarly index of named entities, linked to corresponding mentions in the \textit{Mahānāma} corpus, producing fine-grained annotations for 108,335 entity mentions across 73,632 verses, along with a 5,000-verse expert-verified test set sampled to stress rare mentions. We present the first systematic benchmarking of generative NER against traditional architectures for Sanskrit, finding that fine-tuned generative models perform comparably to task-specific systems. However, all systems show a sharp decline from coarse to fine granularity and struggle with out-of-entity mentions unseen during training. The limitation is not due to data scarcity alone, as fine-tuned models recall unseen entities far worse than seen ones and tend to default to the majority sense under lexical ambiguity.