Databases handle missing data using Bayesian network models
Databases with Missing Values that are Governed by Missingness Mechanisms
Databases
Summary
Databases often have missing information, and this can make it hard to understand or use the data correctly. The authors propose a new way to think about missing data by using Bayesian networks to represent how and why data might be missing. This method moves beyond usual approaches that treat missing values simply as blank or NULL entries. Their approach helps create a probabilistic model of the full database, allowing more informed questions and answers about what the missing data could represent. They also explore when these processes can be computed efficiently and when they become more complex.
What this means in practice
- •For database engineers: Manage and query databases with missing data more accurately by modeling the causes of missing values to improve data reliability.
- •For data integration teams: Improve data fusion systems by incorporating probabilistic models of missing data mechanisms during record linking and merging.
Authors
Leopoldo Bertossi, Farouk Toumani
Abstract
We address the problems of giving a semantics to a relational database (RDB) that has missing values (MVs). The causes for the latter are governed by a Missingness Mechanism that is modelled as a Bayesian Network (BN) that involves the DB attributes as variables. The BN is called a Missingness Graph (MG). Our approach considerable departs from the treatment of RDBs with NULL (values). The combination of the MG and the observed DB allows us to build a block-independent probabilistic DB. We identify two optimal classes of its possible worlds on which QA can be performed. Those classes jointly capture probabilistic uncertainty and statistical plausibility of the implicit imputation of MVs. We obtain tractability results for the computation of some optimal classes; and we also obtain complexity results that characterize the computational feasibility of our approach.