Knowledge graph lakehouse architecture manages enterprise data efficiently
Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale
Databases
Summary
Combining lots of different types of data from many sources is hard, especially when that data is big and fast-changing. The authors propose a new system called Knowledge Graph Lakehouse to handle this better by storing original documents in a flexible storage system and creating smaller, custom knowledge graphs on demand instead of one huge graph. Their design uses cloud technology to keep the system scalable and adaptable to various needs without being tied to any one kind of data. They built a working version and tested it on multiple computers to show it works.
What this means in practice
- •For enterprise data engineers: Build scalable, cloud-based systems that generate context-specific knowledge graphs on demand from heterogeneous enterprise data.
- •For business intelligence teams: Create flexible, application-tailored knowledge graphs to improve data analysis without maintaining a large monolithic graph.
Authors
Bashar Ahmad, Christoph G. Schuetz, Loris Bozzato
Abstract
Modern data architectures for integrating heterogeneous data sources to discover actionable knowledge include the data warehouse, the data lake, and the data lakehouse, which combines the flexibility of the data lake with the structured data management capabilities of the data warehouse through metadata, indexes, and caching. Enterprise knowledge graphs (KGs) offer an alternative approach to such integration for the purposes of data analysis, but handling them at scale must address the Three Vs of big data: volume, velocity, and variety. Contextualized KGs (CKGs) meet part of that challenge by enriching facts with contextual information, yet many applications require only a context-specific subset of that knowledge. We propose the Knowledge Graph Lakehouse (KG Lakehouse), a cloud-native microservice architecture that adapts the data lakehouse paradigm for management of CKGs at scale. The KG Lakehouse indexes incoming source documents along configurable context dimensions and stores these documents in a data lake, from which application-specific CKGs are constructed on demand instead of maintaining a single, monolithic KG. We realize the proposed architecture as a proof of concept, with independently deployable services built from established cloud technology; context dimensions and format-specific document handling enter as configuration, leaving the architecture independent of the application domain. We evaluate the implemented system in experiments on a distributed cluster deployment.