Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval
2026-08-31 • Information Retrieval
Information RetrievalArtificial Intelligence
AI summaryⓘ
The authors studied how to make search systems better at ranking products by improving how they learn from feedback. They found that usual reinforcement learning methods can mess up the way product information is organized, especially when the product list can't change. To fix this, they created a new method called PAO that only adjusts things when it sees clear improvements, keeping the overall structure intact. Tests showed that PAO works better than existing methods on large and public datasets.
dual-encoder retrieverscontrastive similarityrerankersreinforcement learningpolicy-gradientembedding geometrysemantic manifoldfrozen indexgradient updatestopological stability
Authors
Shaowei Wei, Chong Huang, Songtao Fang, Jin Zhang, Zhuojun Wang, Chengfu Huo
Abstract
In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.