Papers for

search infrastructure engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Ginger library delivers faster and smaller graph-based vector search

GINGER: Spicing Up Graph-Based Vector Search

Abstract: We present \textsc{Ginger}, a state-of-the-art open-source C++ library for graph-based approximate nearest neighbor search (ANNS). We developed \textsc{Ginger} at Google to push the speed and memory efficiency of graph search on large and high-dimensional embedding datasets. \textsc{Ginger} surpasses existing ANNS libraries with regards to query throughput and indexing time while matching the memory footprint of the most space-efficient competitors. \textsc{Ginger} builds upon recent advances in proximity graph construction (PiPNN) and vector quantization (TurboQuant). We introduce Nested TurboQuant (NTQ), a two-level quantizer which supports low precision graph navigation and high precision reranking in a single code. This representation achieves near floating point precision with $2.6\times$ less memory. We also compress the size of our proximity graphs by up to $2.5\times$ without impacting query throughput. We combine graph reordering with a new lossless compression codec called AdaptiveGap, which builds upon StreamVByte. At $95\%$ recall, \textsc{Ginger} speeds up query throughput by an average of $1.8\times$ over each dataset's fastest competitor and by at least $2.3\times$ over any single library. This lead grows to $2.9\times$ over compact libraries on high-dimensional embeddings.

Thu 5 NovInformation RetrievalDistributed, Parallel, and Cluster Computing
The gist
Finding similar items quickly in large datasets is hard when the items are described by many numbers. The authors developed Ginger, a new software library that speeds up search by organizing data points as a graph and compressing them efficiently. Their new ways of compressing and navigating the graph let computers find good matches faster and using less memory. Ginger outperforms current tools on both speed and storage while maintaining search accuracy.
Open → 2611.08306v1

Graph compression cuts memory for fast nearest neighbor search

Lossless Compression of Proximity Graph Indexes for Efficient Approximate Nearest Neighbors Search

Abstract: Graph-based Approximate Nearest Neighbor (ANN) indexes, such as HNSW, are the backbone of modern vector search. However, their massive memory footprint creates a severe infrastructure bottleneck. While vector quantization is widely adopted to reduce the memory required to store data vectors, the graph's edge structure increasingly dominates total memory usage. Compressing these adjacency lists for online search presents a critical system challenge: because decompression occurs during query processing, i.e., while navigating the graph, memory savings cannot come at the expense of query latency. In this paper, we present a compression technique explicitly tailored for live HNSW routing. By applying offline graph permutations to artificially induce structural locality, we enable the use of SIMD-accelerated integer compressors, such as Stream VByte, for rapid on-the-fly decompression. Evaluated across diverse real-world workloads, we significantly outperform the current state of the art in both compression ratio and query latency. Our experimental results show that we achieve compression ratios up to 52.7% with negligible penalties in latency compared to the uncompressed baseline. Furthermore, our technique has been effectively deployed in production at eBay, achieving up to 44.3% reduction in edge storage with no degradation in query latency.

Tue 27 OctInformation Retrieval
The gist
Finding the closest items in large collections often uses graphs that connect similar things, but saving these graphs takes up a lot of computer memory. The authors developed a way to shrink the memory used by these connection lists without slowing down how quickly searches happen. They rearranged the graph data offline so it compresses better and can be quickly unpacked when needed. Tests show their method saves over half the memory with almost no delay, and it’s already used at eBay to reduce storage without affecting speed.
Open → 2610.37176v1

Generative retrieval improves item suggestions while reducing search costs

PinGER: Token-Efficient Generative Retrieval with Semantic IDs

Abstract: Generative retrieval replaces nearest-neighbor search with token-by-token decoding of an item's semantic ID (SID). But existing SID-based methods primarily use L levels of tokens sequentially to represent each item in the input history. As a result, a fixed context window holds L times fewer items than it would with one embedding per item. We present PinGER, a generative retriever deployed at Pinterest that retains SID outputs while making input histories compact, built on three ideas. (1) An asymmetric attention mask over interleaved SID and dense tokens lets later items attend only to the dense ones, so the model is supervised at every SID position in training yet serves one token per item while computing the same function. (2) Generation is controllable through goal-conditioned outputs, reinforcement learning, and a prefix-diversity penalty, which together steer retrieval toward surface-specific goals and diverse candidates. (3) A serving stack built for efficient wide-beam generation is paired with decoding constrained to valid identifiers and candidate resolution from a host-local index. Deployed in Related Pins, PinGER lifts saves of fresh Pins by 3.60% and overall saves by 0.62%, while reducing the surface's end-to-end latency by 0.7% and eliminating the cost of the nearest-neighbor search it replaces.

Sat 24 OctInformation Retrieval
The gist
Finding related items online often relies on searching through many possibilities, which can be slow and costly. This paper presents PinGER, a way to represent items compactly so a computer can generate identifiers for them step-by-step instead of searching through all items directly. The authors show that this method, used at Pinterest, makes the system faster and cheaper while improving how often users save related items. They achieve this by combining efficient input handling, controls over what the model generates, and a smart serving setup.
Open → 2610.33435v1

Gating skips reranker calls for faster multi-hop retrieval with small accuracy loss

Per-Query Gating of LLM Rerankers for Multi-Hop Retrieval

Abstract: LLM rerankers add of the order of \$0.2-0.3 per 1,000 queries and about a second of tail latency on top of a graph-augmented dense pipeline such as HippoRAG2, and on three multi-hop benchmarks they improve final-hop top-K coverage on seven of nine (dataset, K) cells, by up to +34.8 pp. We ask whether a learned per-query gate can skip the reranker where it will not help, using only features available before the LLM call (27 score and lexical statistics of the two retrieval lists plus a PCA of a small query embedding) with an executable fallback. Every choice, including the fallback and the threshold, is made inside the training fold and applied once to held-out queries, and harmful skips (the rerank would have found the target, the fallback did not) are reported next to the aggregate coverage. Across nine cells on 2WikiMultiHopQA, MuSiQue and HotpotQA the gate skips 51% of calls at an average held-out LastHop@K cost of 1.2 pp; four cells meet a pre-registered 1 pp rule, harmful skips occur in eight (190 harmful against 136 beneficial), and a random gate at the same skip rate loses 2 to 11 pp on the high-lift cells. A second rule sets each cell's threshold from a pre-specified budget on the expected harmful-skip rate over Platt-calibrated harm probabilities (ECE 0.025 after calibration, 0.094 before): at a 1 pp budget the gate skips 42% at -0.8 pp with 66 harmful skips and six cells within 1 pp, but realised harm exceeds the promise in six cells (mean 1.45 vs 0.83 pp), a selection optimism we quantify; a 0.5 pp budget realises about 1 pp. The harm probabilities are calibrated but barely discriminative (AUC 0.16 to 0.70). An earlier version reported 73% "lossless" savings; that figure rested on an oracle fallback and a wrong MuSiQue target, and we document both.

Sat 19 SeptInformation RetrievalArtificial IntelligenceComputation and Language
The gist
Large language model (LLM) rerankers improve search accuracy but slow down the system and add cost. The authors develop a gate that decides for each query whether the reranker is needed, based on quick-to-get features before running the costly reranker. This gate skips about half of reranker calls, saving time and money with only a small drop in accuracy. The authors carefully measure harm caused when the gate skips useful reranking and adjust thresholds to control this risk. The gate’s predictive power is limited, so some harmful skips remain despite calibration.
Open → 2609.22880v1

Mixture-of-experts language models improve search speed and accuracy

Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

Abstract: Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, evaluating them across diverse datasets, and measuring query encoding time under the same serving configuration. We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR. One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time. We further show that the number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%. Recent rerankers provide only modest additional gains over strong MoE first stages, which often match or exceed the reranked configurations we evaluate. Together, these results show that MoE LLMs can be strong and efficient first-stage retrievers.

Fri 11 SeptInformation RetrievalComputation and Language
The gist
Searching through large amounts of text quickly and accurately is hard because big language models take a lot of computing power. The authors studied a special kind of model called mixture-of-experts (MoE) that only uses part of its brain when answering, making it faster. They found these MoE models can find relevant information better and more efficiently than regular models of similar size. Also, you can reduce how much of the model is active during search without losing much accuracy. This means MoE models can make searching smarter and quicker at the same time.
Open → 2609.13486v1

Compact binary codes improve document search efficiency and accuracy

Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval

Abstract: Retrieval-augmented generation (RAG) depends on dense retrieval: each document is stored as a learned vector, and a query is answered by finding its nearest neighbors in that vector space. Keeping one full-precision vector per document is the dominant index cost at corpus scale, so retrieval systems replace each vector with a short code of a few bytes---a step called quantization. Standard quantizers such as product quantization (PQ) pick the code that reconstructs the original vector most closely. A single code is even more useful if it serves several byte budgets at once: when its short prefixes are each directly searchable, a deployment can set its efficiency--quality operating point without re-encoding the corpus. But training all prefixes under one objective makes the early bits a compromise across budgets---short codes improve while the full-width code degrades. Quantization to low-bit representation, such as binary codes, further sharpens the conflict. We introduce Matryoshka Hash Representations (MHR), a two-stage procedure that separates full-width training from prefix organization. MHR first learns a longer binary code, then freezes the model and trains additional zero-initialized residual code adaptors for directly searchable prefixes. Documents are stored at one bit per coordinate, while queries keep continuous logits like PQ to attain sufficient expressivity. We implement the search process with FAISS FastScan. Trained on MS MARCO and zero-shot transferred to seven BEIR datasets, MHR reaches .5561 NDCG@10 and .6535 Recall@100 at 32 bytes, surpassing the best baseline of the same budget. The advantage is more pronounced in lower budgets. The same code also strengthens two common pipelines: shortlisting candidates for full-precision reranking, and pruning a low-storage graph index such as LEANN.

Mon 7 SeptInformation RetrievalArtificial IntelligenceMachine Learning
The gist
Searching large collections of documents can be slow and costly when storing detailed information about each document. The authors introduce a new method called Matryoshka Hash Representations that stores document data in compact binary codes which can be searched quickly without losing much accuracy. This method first learns a longer binary code, then creates shorter subsections of that code that can be searched efficiently. Their approach works better than current techniques, especially when very little storage space is available.
Open → 2609.07276v1