Dhārā bench measures search for questions on indian knowledge texts
Dhārā Bench: An Evidence-Oriented Retrieval Benchmark for Indian Knowledge Systems
Information RetrievalComputation and Language
Summary
Finding reliable information in ancient Indian texts can be tough. The authors created Dhārā Bench, a test with 325 questions and thousands of text pieces from Indian Knowledge Systems. They measured how well a standard search method, BM25, could find relevant passages to answer these questions. The results reveal strengths and weaknesses in retrieving different types of text, especially verses, and show gaps between identifying needed documents and fully supporting answers. They made the data and tools publicly available to help improve search in these texts.
What this means in practice
- •For natural language processing teams: Evaluate and improve algorithms for searching and understanding complex questions over Indian Knowledge Systems texts using a standardized test set.
- •For digital library developers: Develop search features tailored to Indian classical and traditional knowledge documents using detailed question-to-text relevance annotations.
Authors
Avinash Holenarasipur Nagendra, Anand Kaniyaar Sriranga Prasad, Kousik Sarathy Sridharan, Mohan Raghavan
Abstract
Dhārā Bench is a retrieval benchmark for Indian Knowledge Systems (IKS), with 325 questions in 12 categories and 60,121 text chunks from 15 source groups. The categories were developed based on 4,500 questions submitted to the Dhārā application; the evaluation set combines corpus-aligned application queries with expert-contributed questions. The annotations comprise 7,578 question-chunk relevance links involving 6,847 distinct corpus chunks. For 34 composite and multi-hop questions, 401 conjugate groups specify alternative sets of passages that jointly support each question. Chunk identifiers and hierarchical references connect the judgments to their source texts. The paper describes corpus preparation, annotation, and validation, and reports a fixed BM25 baseline implemented with Pyserini with $k_1=0.9$ and $b=0.4$. Across all questions, BM25 achieves MRR@20 of 0.315, Recall@10 of 0.182, and nDCG@10 of 0.205. On the group-annotated subset, complete-evidence scores are CE_Recall@20 = 0.294 and CE_MRR@20 = 0.141. The category results expose particularly weak verse retrieval, while group scores reveal gaps between finding individual passages and completing their support. The corpus and evaluation code are available at https://github.com/Avinash2236/Dhara-Bench.