Bottom-up clustering improves semantic ids for generative retrieval
Exploring Bottom-Up Clustering for Creating Semantic IDs
Information RetrievalArtificial Intelligence
Summary
Finding unique and meaningful identifiers for items helps search engines retrieve better results. This paper shows how using a method called bottom-up clustering can create semantic IDs that uniquely identify products while keeping their meaning intact. The researchers present a way to group items based on their similarities, which improves how these IDs capture useful information. These improved semantic IDs help systems that generate search results perform better.
What this means in practice
- •For search engine developers: Create better product identifiers that improve generative search accuracy by preserving item similarities.
- •For digital marketplace engineers: Enhance item cataloging systems by generating unique and semantically meaningful product IDs that improve retrieval tasks.
Authors
Leah Woldemariam, Sudhanshu Garg, Taha Belkhouja, Charles Kim-Yip, Ali Sahami
Abstract
The success of generative retrieval has largely been attributed to the use of Semantic IDs, which improve over arbitrary item-level identifiers such as hashes by capturing the semantics of items. The main challenges faced when constructing Semantic IDs, however, is in mapping each identifier to a unique product and capturing information valuable to downstream tasks. Past works have appended additional codewords to de-duplicate item identifiers and utilized residual quantization to create hierarchical clusters. In this work, we present an algorithm for generating Semantic IDs that ensure the identifiers are both unique and preserve the structure of the original embedding. Key to our work is the use of bottom-up clustering to preserve local structure in the embedding space, improving the clustering quality of the resulting Semantic IDs and their utility for downstream generative retrieval.