Large language models produce effective search embeddings with few examples
Effective Dense Retrieval using Only In-Context Examples
Information RetrievalComputation and Language
Summary
Finding relevant information quickly often requires special training on search methods. This paper shows that large language models can create good search-friendly representations just by being given a few examples in their input, without extra training. The approach, called RICE, uses these examples to help the model understand how to encode questions and documents so they match better. This method improves search accuracy while being simpler because it doesn’t need retraining the model.
What this means in practice
- •For search engine developers: Generate better document and query embeddings for search engines without retraining models, improving relevance with just few example prompts.
- •For chatbot developers: Enhance chatbot knowledge retrieval by producing effective embeddings from language models using only in-context examples, enabling faster deployment.
Authors
Nour Jedidi, Abdul Basit Ali, Hang Li, Jimmy Lin
Abstract
Turning decoder-only large language models (LLMs) into strong dense retrievers typically requires some form of retriever training. In this paper, we ask whether LLMs can instead be prompted to produce effective representations for dense retrieval given only a few in-context examples. To answer this, we introduce RICE (Representations from In-Context Examples), a simple "training-free" approach that extracts high-quality dense representations from LLMs. To do so, RICE conditions the LLM on examples that provide a shared context for query and document encoding. Our results demonstrate that RICE embeddings can substantially improve the accuracy of prompt-based LLM embeddings, establishing it as a simple method to build LLM-based dense retrievers that do not require training. We release our code at https://github.com/nourj98/RICE.