Data agents transform data systems with autonomous smart processing
Data Agents: Agentic Data Systems
DatabasesArtificial IntelligenceComputation and LanguageMachine Learning
Summary
Traditional data systems struggle to handle complex and varied information without lots of human help. The authors propose data agents, which are like smart assistants that organize, understand, and manage data mostly on their own. These agents work proactively and learn from feedback to improve how they handle data. The paper shows that these smart data agents perform better than current methods on real tests. This approach could change how data teams work by reducing manual effort and making smarter decisions.
What this means in practice
- •For data engineering teams: Manage and automate complex data workflows with minimal manual design and intervention.
- •For business intelligence teams: Automatically interpret and analyze heterogeneous data sets to improve insight generation.
Authors
Guoliang Li, Peiyao Zhou, Xuanhe Zhou, Ji Sun, Yuyu Luo, Ju Fan
Abstract
Traditional data systems face profound limitations in the AI era, relying on human-crafted pipelines, lacking semantic understanding of heterogeneous data, and operating through rigid, reactive processing. To address these challenges, we propose a new paradigm called the Data Agent, designed to manage, process, and analyze data with minimal human intervention. Data agents autonomously execute a wide range of data-related tasks, transforming traditional data systems by shifting from manual design to autonomous orchestration, from literal manipulation to semantic interpretation, and from reactive to proactive processing. Our Data Agent system includes six components: semantic data organization, semantic operators, agentic pipeline orchestration and optimization, feedback-driven refinement, memory management, and proactive adaptation. Building on this foundation, we also develop two specialized agents: the data analytics agent and the data science agent. Experiments on real benchmarks demonstrate significant performance gains of our data agent over state-of-the-art methods. We identify open challenges to guide future research in building fully autonomous data systems.