Query context speeds up transformer models inside databases
QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing
DatabasesArtificial Intelligence
Summary
Predictive query processing uses AI models to improve database tasks, but typical methods only see a small part of the data, missing useful query details. The authors present QCATS, which looks at the whole query and related statistics to smartly select parts of the AI model to run, making inference faster. This method reduces processing time by up to 4.4 times without losing accuracy. QCATS also introduces computer system tricks to handle processing better between CPUs and GPUs.
What this means in practice
- •For database engineers: Improve predictive query response times by using query-aware AI model slicing within the database engine.
- •For cloud infrastructure teams: Deploy more efficient AI-assisted database services with reduced AI model inference latency using query context information.
Authors
Yueying Li, Zhongle Xie, Ke Chen, Lidan Shou
Abstract
In-database predictive query processing increasingly applies Transformer-based models within relational pipelines. However, existing in-database inference typically exposes only tuple-level model inputs to the inference runtime, leaving relational predicates and metadata statistics invisible to neural execution planning. In this paper, we propose QCATS, a query context-aware transformer slicing framework that enables efficient sparse inference inside database systems. QCATS executes at query granularity: instead of routing individual tokens or tuples during inference, it uses query predicates and metadata statistics to pre-select context-aligned FFN slices before model execution. The framework comprises offline expert construction and lightweight query-level routing that dynamically selects experts during execution. QCATS further introduces system optimizations, including asynchronous CPU-GPU pipelines and routing-aware batching. Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42x latency reduction while preserving prediction accuracy comparable to dense baselines.