Papers for

digital library engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Dhara engine improves search in Indian Knowledge System texts

Dhara: Evidence-Oriented Retrieval for Indian Knowledge Systems

Abstract: Relevant evidence in Indian Knowledge Systems (IKS) collections may be distributed across passages in different parts of a work, including verses, translations, and commentaries. Locating it requires matching natural-language questions and partially remembered verse fragments to source passages. We present Dhara, a retrieval engine combining hybrid search, chapter- and book-level Entity Notes (EN), Fuzzy Verse Retrieval (FVR), and neural reranking. Five comparisons on the 325 questions in Dhara Bench examine retrieval representations against BM25, EN's contribution, reranker quality and computation time, FVR, and Gemini. BGE-M3 single-vector retrieval exceeds BM25 in nDCG@10, Recall@20, and MRR@20 across all categories. Its scores are broadly comparable to ColBERTv2, with mean retrieval times of 0.108 s and 0.962 s under the same hardware configuration and a smaller memory footprint for chunk storage on disk. Adding EN to a direct branch with a matched candidate count improves nDCG@10 in all twelve categories with Qwen and ten with BGE; the combined ranking is capped at the same candidate count. Qwen improves nDCG@10, Precision@5, and MRR@20 over BGE in every category, at approximately 6.72 times the overall estimated serial computation time. FVR increases Verse-category nDCG@10 by approximately 20 percentage points with either reranker. Gemini is evaluated after source mapping and manual review with relevance annotations visible. The Qwen pipeline with EN and FVR has higher Recall@20 than Gemini in ten categories and stronger complete-evidence retrieval in Multi-hop questions, while Gemini retains early-relevance advantages in several categories. The results inform retrieval choices according to evidence requirements and resource budgets.

Sun 8 NovInformation RetrievalComputation and Language
The gist
Finding relevant information in ancient Indian texts is hard because important details can be scattered across verses, translations, and commentaries. The authors created Dhara, a search tool that combines several techniques to match user questions with these scattered pieces of text. They tested Dhara against existing search methods and found it recovers more relevant results faster and with less computer memory. This helps users locate evidence more effectively in these complex collections.
Open → 2611.11197v1

Challenges and methods for retrieving domain specific information

Domain-specific information retrieval

Abstract: Information Retrieval (IR) is much more than ad hoc web search: a large portion of the information seeking activities takes place in specific domains, often addressing work-related tasks. Examples are lawyers finding relevant prior cases for a legal case and academic researchers studying the background material for a paper. In Chapter 2 we outline the most common domain-specific tasks, datasets and applications. Domain-specific IR has a number of challenges that make set it apart from simple web search tasks: they often deal with long documents, long queries and long sessions with specific user needs. We discuss these in Chapter 3. We also address issues related to bias and fairness of models. In Chapter 4, we describe the models and methods developed for domain-specific tasks and data, addressing the specific challenges, including user aspects related to domain-specific IR tasks, dealing with queries, user models, and user interactions. We show that many of the methodological advances in domain-specific IR are transferable to other domains and even to open-domain web search. We conclude with the main takeaways and propose research directions for the near future.

Thu 5 NovInformation Retrieval
The gist
People often search for information in specific professional or academic areas, not just on the general web. The authors explain that such searches can be more complicated because they involve long documents, detailed queries, and extended user needs. They review the unique challenges in these searches, including fairness and bias in search models. Their work also covers the models developed to address these issues and highlights that advances in specialized searches can help improve general web searches too.
Open → 2611.08228v1

Scientific paper names cause confusion across research fields

"PRISM" Is Not All You Need: Cross-Topic Name Collisions and Scientific Paper Retrieval

Abstract: Scientific names increasingly connect papers from different research topics, leaving readers to recover a paper's identity from context. We study this convergence in 564,861 AI-related arXiv records. Cross-topic name collision grows 4.33-fold from a pooled 2020-2022 reference to 2026, compared with 1.12-fold within topics. A second sentence encoder reproduces the pattern, and name-group resampling quantifies its uncertainty. Fixed word-frequency bands locate the main increase in an intermediate-frequency vocabulary shared by many research areas. We then measure the ambiguity that remains after supplying research clues. With the same three-term query construction, shared-name targets retain matched identification deficits of 25.7 percentage points under title-based BM25 and 6.9 points under title-plus-abstract E5. Within fixed same-name candidates, listwise language-model ranking reaches 89.2% using the query and candidate titles. Topic-specific comparisons show that context resolves more cross-topic ambiguity. The evaluation separates candidate competition, document fields, and contextual ranking through lexical, dense, and language-model baselines. Source-linked cases and a portable evaluation collection make the phenomenon inspectable and reusable for scientific retrieval and research assistants.

Sun 1 NovInformation RetrievalDigital Libraries
The gist
Scientific papers often use names that appear in multiple research areas, making it hard to find the right paper just by its name. The authors studied over half a million AI papers and found this name confusion is growing much faster between topics than within a single topic. They tested different search methods and found that adding context like abstracts helps reduce confusion but doesn’t eliminate it entirely. Their work provides tools and data to help improve how scientific papers are found despite these naming overlaps.
Open → 2611.01362v1

Weighted clustering improves large biomedical citation network analysis

Similarity-Weighted Leiden-CPM Clustering of a 7.4-Million-Node Biomedical Citation Graph: Weight Alignment, Scale, and Sparsity

Abstract: Clustering citation networks into research communities supports science mapping, research evaluation, and the analysis of funding portfolios. Biomedical articles also carry metadata, such as MeSH terms, publication year, and abstracts, that can be turned into similarity-based edge weights for Leiden clustering under the Constant Potts Model (CPM). Studies of hybrid relatedness compare the accuracy of such clusterings, but how the weights interact with the CPM resolution parameter on a large graph is less clear. Here it is shown that, on a citation network derived from the cell biology research literature (7.4 million nodes, 57.5 million edges), the effect of the weights is governed mainly by their scale and by edges of weight zero, and that, once these are controlled, weighting gives modestly more coherent clusters. Edges were weighted with seven topological and semantic similarity features and clustered with Leiden-CPM, using a method updated to fix a bug in how earlier code attached weights to edges. Mean weights of 0.0015 to 0.23 raised the effective resolution 4- to 650-fold; dividing by the mean restored most of the coverage, except for topological weightings, in which the 47% of edges with weight zero hold most of it. At equal coverage, weighted clusters were more coherent than unweighted ones, also on features outside the weight. For weighted CPM clustering, weights should therefore be attached to edges by key, normalized or reported with their effective resolution, and checked for zero-weight edges.

Sat 31 OctSocial and Information Networks
The gist
Clustering scientific articles helps us map research areas and evaluate funding, but biomedical articles have extra data like keywords and abstracts that can help measure similarity. The authors studied how attaching these similarity measures as weights affects clustering on a very large citation network of cell biology papers. They found that the scale of these weights and how zero-weight edges are handled mostly control the cluster results. When these factors are managed properly, the weighted clustering produces groups of articles that are slightly more coherent than unweighted approaches.
Open → 2611.01049v1

PubMed citation graphs enhanced with article similarity at large scale

Building Weighted PubMed Citation Graphs at Scale: A Pipeline and an Audit of Edge-Similarity Features

Abstract: Studies of scientific literature often combine citation links with article metadata, but the data preparation needed to do so at the scale of millions of papers is laborious and error-prone. This paper describes a pipeline that turns PubMed metadata and a citation network into a graph whose edges carry seven similarity features: bibliographic coupling and co-citation (each as a count and as a Jaccard coefficient), publication-year proximity, MeSH-term similarity, and similarity of MedCPT title-abstract embeddings. Applied to a citation network derived from the cell biology research literature, the pipeline yields 7,445,898 nodes and 57,512,086 edges with complete year, MeSH, title, and abstract metadata. The paper reports how nodes are matched across sources, how duplicate identifiers are resolved, and which intermediate tables grew to hundreds of gigabytes. An audit of the full feature table shows that 47.2% of the edges share neither references nor citing documents, so their topological features are zero, and that the MeSH feature is nonzero on only 0.28% of edges. Where both kinds of signal exist, they agree: edges with higher coupling also have more similar abstracts and closer publication years. In a pilot, replacing the median by a mean or best-match aggregation makes the MeSH feature nonzero on nearly all article pairs, and near-identical texts point to possible duplicate records that identifier checks miss. The parsing, graph-construction, and feature code are public; the paper states which data are released and which are not. A companion paper uses the resulting weighted graph for clustering.

Sat 31 OctSocial and Information Networks
The gist
It is hard to study millions of scientific papers and their connections because preparing the data correctly can be very difficult and can lead to mistakes. The authors created a computer pipeline that builds a huge network of papers from PubMed, linking them not just by citations but also by features like similar topics and abstract texts. This network has millions of papers and tens of millions of links, each carrying detailed similarity information. The authors also checked how well these similarity features work and found some interesting patterns about which papers are connected and how similar their content is.
Open → 2611.01044v1

Decision models do not improve reranking over standard fine-tuning

System One models Are Not Better Rerankers

Abstract: System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.

Fri 23 OctComputation and LanguageInformation Retrieval
The gist
The paper compares two ways to improve search ranking models: decision fine-tuning, which calibrates probability answers, and typical reranker fine-tuning. The authors find that decision fine-tuning does not make models better at reranking results compared to standard approaches. This holds true across different model sizes and testing collections. They also show that better calibration of decisions does not compensate for poorer ranking quality. Overall, fine-tuning a reranker remains the best choice for improving search rankings.
Open → 2610.32318v1

Vision language teacher improves visual document retrieval accuracy

What Transfers from a VLM Teacher? Comparing Supervision Signals for Visual Document Retrieval

Abstract: Visual document retrievers are trained contrastively: each query is matched to one page labelled relevant - the positive - and pushed away from negatives, pages presumed irrelevant. Recent methods distil a vision-language model (VLM) teacher into the retriever by enriching that positive, transferring the teacher's attention over it or a description of it. We ask whether the teacher is better spent on the other side, judging the candidates the retriever mines as negatives, which the label says nothing about. With student, data, optimizer and evaluation fixed, teacher-judged hard negatives and score distillation raise ViDoRe v2 nDCG@5 from 55.2 to 62.6 and 63.0; description alignment, as adapted here, gains 2.6 points and attention grounding nothing measurable. Against teacher-free rules that select four candidates from the same mined pool at identical training compute, the best of which is the positive-aware threshold current systems use, the teacher's judgement adds 4.1 points on v2 and 1.7 on v3. This is consistent with how incomplete the labels are. Annotators judge about two of a query's four top-ranked mined candidates relevant, none of them labelled, so training pushes the retriever away from relevant pages treated as negatives. What reaches the student is coarse: under a greedily decoded 0-100 rating prompt, 82% of the teacher's ratings come back at one end of the scale or the other, and a relevant/irrelevant partition keeps most of the distillation gain. A ten-annotator audit places the teacher within the range of variation among human annotators, and finds it reliable where a query has a single determinate answer. We release the code, the teacher's 3.3M judgements and page descriptions, the mined pools, the human audit and the trained adapters at https://github.com/elastic/vdr-teacher-signals.

Tue 6 OctInformation Retrieval
The gist
Finding the right document page for a query is hard when only limited correct answers are known. The authors show that using a vision-language model (VLM) teacher to better judge which pages are truly relevant helps retrievers learn more accurately, especially by identifying tricky false negatives. They compare different ways the teacher’s signals can guide learning and find that judging negative candidates is more effective than enriching positive examples. The teacher’s judgments align well with human annotations, especially when there is a clear single answer, improving retrieval performance notably.
Open → 2610.09177v1

ConRAG improves retrieval of multi-step entity connections with less computing cost

ConRAG: Lightweight inference of multi-hop relations

Abstract: Understanding how two entities are connected often requires tracing multi-hop relations across documents to identify intermediate entities and supporting evidence that explain a connection. This is a task that appears frequently in scientific research and other knowledge-intensive analyses. We formalise this setting as multi-hop relation inference: given two known endpoint entities, we aim to recover the bridge entities and evidence-grounded reasoning chains that connect them across a document corpus, and to generate an explanation grounded in the retrieved evidence. Existing multi-hop RAG systems typically seek an unknown answer entity rather than explicitly recovering the connection between two known endpoints and graph-based approaches often rely on costly LLM-extracted knowledge graphs that limit scalability to large document collections. We introduce ConRAG, which builds a lightweight entity-document graph from entity co-occurrence and LLM-based entity filtering. Its connective retrieval infers and semantically ranks paths between two endpoints. On MuSiQue and 2WikiMultiHopQA, ConRAG consistently improves bridge entity and reasoning chain recovery over strong RAG baselines, while reducing graph-indexing token cost by up to roughly 1.5 orders of magnitude. Our results show that endpoint-constrained path retrieval provides an effective and index-efficient approach to evidence-grounded relation discovery.

Mon 28 SeptMachine Learning
The gist
Sometimes to understand how two things are related, you need to find other things in between that connect them. The authors call this multi-hop relation inference. They created ConRAG, a method that makes it easier and faster to find these connecting steps and explain the link between two known things using lots of documents. ConRAG uses a simple graph showing where entities appear together and filters them smartly to find the best path. It works better and uses much less computing power than older methods on two test sets.
Open → 2609.35193v1

Visual document retrieval improves by picking tokens after query arrives

Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrieval

Abstract: Late-interaction visual document retrievers preserve fine-grained page evidence by storing many token embeddings per page, but the resulting storage and query-time interaction costs make large-scale deployment expensive. Pooling document tokens before indexing offers a natural remedy, yet static pooling must decide which visual evidence to preserve before the query is known. We study an alternative: a heavily compressed hot-path index generates candidates, after which query-aware token budgeting operates on the original token sets of the shortlisted pages. We formulate this stage-two selection as a budgeted MaxSim coverage problem, show that a clipped version is monotone submodular, and compare coverage-only, cluster-guided, token-wise, and marginal-gain policies. On ten ViDoRe tasks with ColModernVBERT, direct static pooling reduces macro normalized discounted cumulative gain at rank five from 0.6309 without compression to 0.4738 at a thirty-two-fold pool factor. Under the same candidate-generation regime and a pool-factor-eight-equivalent reranking budget, token top-k recovers 93.93 percent of the full-token score, while greedy marginal-gain selection recovers 98.39 percent. Held-out and leave-one-dataset-out evaluations yield positive greedy improvements over token top-k on every dataset. The latency analysis reveals two useful operating points: token top-k for interactive retrieval and the naive greedy implementation as a quality upper envelope. Together, these results show that late-interaction visual retrieval benefits from query-aware allocation rather than query-agnostic pooling alone.

Mon 7 SeptInformation RetrievalArtificial Intelligence
The gist
Finding relevant images of documents quickly is hard because pages store lots of detailed information, which slows things down and uses lots of storage space. The authors study a two-step method: first, they use a small index to pick some promising pages, then they carefully choose which pieces of those pages to look at based on the actual search query. This method chooses tokens (small units of information) smartly, which keeps the search accurate but much faster. Their tests show this adaptive approach works better than just compressing data without considering the query.
Open → 2609.07262v1