VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors describe VersaDB, a new database system designed to better store and manage AI training datasets that have different types like text and images. Unlike existing tools, VersaDB organizes data efficiently using a special storage layout and fast index trees, helping speed up access to data. It also breaks data into smaller parts automatically and keeps track of metadata to improve performance. Their tests showed VersaDB can make data processing over five times faster and works well even when using many processors at once.
AI datasetsmultimodal datadatabase storageB+ tree indexshardingmetadata managementdata layoutparallel processingTFRecordGPU acceleration
Authors
Cong Wang, Zelin Liu, Yang Luo Ran Zhang, Zhijian Guo, Hui Zhang, Fan Yu, Yanfei Cao, Naijie Gu, Jun Yu
Abstract
The AI field has been rapidly developing, leading to the emergence of a large number of AI training datasets of various types. These datasets contain different modalities, including text, images, audio, etc., and may come in various data storage formats. With the advancement of AI hardware, AI computation units like GPUs, TPUs, and NPUs can greatly accelerate the training speed of AI models, which in turn increases the demand for faster data processing. When using existing AI processing frameworks to handle datasets with different modalities and storage formats, processing speeds may be suboptimal due to issues such as data layout and the way users handle the data. Therefore, using a unified database to store multiple data formats can better manage and optimize data access. In this paper, we introduce VersaDB, a database designed specifically for AI datasets with various modalities. We implemented a page-based storage system, separating structured and unstructured data. Additionally, we generated B+ tree-based index files to accelerate data access. VersaDB supports automatic sharding and maintains a hierarchical metadata management system, with corresponding metadata maintained at the page, shard, and global levels, forming the foundation for the efficient operation of the database. We also focused on ease of use by providing APIs for directly converting datasets into VersaDB, as well as APIs for converting popular AI data storage formats (e.g., CSV, TFRecord, .bin) into VersaDB.Our experiments show that using VersaDB can achieve up to 5.35x acceleration and maintain consistent performance across different parallelism levels.