AI summaryⓘ
The authors describe UBASE, a system used by ByteDance since 2016 that evolved from a text search engine into a powerful search tool combining AI features like vector retrieval and filtering. They highlight two main challenges when dealing with massive data: the heavy memory needed to build indexes and the high cost of storing these indexes entirely in memory. To solve this, the authors introduced a new quantization method allowing efficient and accurate index construction using less memory, and a flexible storage engine that balances memory use and speed by mixing memory and SSD storage. Their improvements enable faster processing, much lower memory use, and reduced costs while handling extremely large datasets in real-world settings.
UBASEvector retrievalquantizationmemory optimizationindex constructionSSD storageAI search systemshigh-dimensional vectors
Authors
Yao Tian, Yuncheng Lu, Liyao Xiong, Yuming Xu, Hao Zhang, Weichen Zhao, Xi Zhao, Bo Kuang, Dongyu Wang, Jiehui Li, Yakun Li, Lei Zhang
Abstract
Since 2016, UBASE has been the foundation of ByteDance's search infrastructure, scaling to more than 7,000 clusters and 300 PB of indexed data. Driven by the demands of AI workloads, UBASE has evolved from a text search engine into a unified AI search system supporting vector retrieval, lexical matching, and predicate filtering. Its largest deployment indexes nearly one trillion high-dimensional vectors. This scale exposes two central bottlenecks in AI-era retrieval: memory-intensive graph-index construction under sustained ingestion, and the prohibitive cost of keeping vector indexes entirely in memory. UBASE addresses these bottlenecks with two techniques. First, it introduces a quantization-aware vector kernel based on SymRaBitQ, a new symmetric quantization scheme with tight theoretical guarantees that allows index construction to run directly in the quantized space accurately and efficiently without retaining a copy of full-precision vectors. Second, it provides a hybrid storage engine that supports memory-resident, hybrid, and SSD-resident deployments, with fine-grained record-level caching to trade memory for latency under operational control. On large-scale benchmarks, UBASE improves throughput by up to 3x, reduces indexing memory by 80%, and lowers operating cost by 86% compared with prior systems, while supporting trillion-vector scale, write-heavy or latency-sensitive workloads in production.