Evolving semantic indexes improves search and recommendation quality
Evolving Semantic Indexes for Generative Retrieval
Artificial IntelligenceInformation RetrievalMachine Learning
Summary
Search engines and recommendation systems can work better by representing information with special codes called semantic identifiers. These codes are tricky to design because their parts interact, and improving them by hand is hard. The authors used a clever process where an AI program tries out many small changes to these codes and learns which ones work best. This method helped improve search accuracy on several datasets without relying on costly AI services. It shows that evolving parts of the system with fast testing can make search smarter.
What this means in practice
- •For search engine engineers: Improve search accuracy by evolving discrete semantic codes without retraining full models repeatedly.
- •For e-commerce platform developers: Boost recommendation quality on multilingual e-commerce data through evolved semantic index components.
Authors
Jinming Nian, Jyun-Yu Jiang, Henry Tsai, Yi Fang, Zhirong Liang
Abstract
Generative retrieval (GR) casts search and recommendation as autoregressive generation of discrete semantic identifiers (SIDs). SID design largely determines GR quality, but its discrete, interdependent components (SID construction, collision handling, decoding, and training) cannot be optimized end-to-end with gradients and remain largely hand-designed. We introduce a framework that improves these components through LLM-driven evolutionary program search, to our knowledge the first application of this approach to GR. A locally served, open-weight 31B model mutates executable GR components, so no proprietary LLM API is used, and a 1M-parameter proxy decoder provides fast feedback in place of repeatedly training the 1B target generator. Across 21 SID programs on NQ320K, proxy and target Recall@100 show strong rank correlation (Spearman $ρ= 0.90$). Guided by this proxy, evolution improves the seed GR on NQ320K, MSMARCO300K, and a multilingual e-commerce dataset under a fixed training recipe. On NQ320K, jointly evolving SID construction and collision handling improves Recall@1 from 55.8% to 59.3% and Recall@100 from 85.8% to 90.6%. A component study shows that proxy reliability depends on what is evolved: improvements transfer reliably for the semantic index but not for the generator's training objective or the full pipeline. LLM-driven evolutionary search is thus a practical tool for GR design, provided proxy reliability is established for the chosen search space.