Directed bipartite Hypergraph: Representation of data edits for constraint-based data cleaning

Abstract

Constraint-based data cleaning captures data violations to a set of constraints called data quality constraints. Data edits is one of constraint type besides integrity constraint that used for checking data inconsistencies which come from census or survey questionnaire (questionnaire schema). Data edits contain some variables and describe their relationship using AND and OR operator. The relationship needs to be represented in a structure that can find the best data repair solution. Graph is a generic structure to represent a relationship. In previous studies, hypergraph is used as a solution to represent variable relationships of the violated integrity constraint. Such solution is not efficient for data edits. Hypergraph cannot show the relationship between data edits as a whole. This can trigger more new errors. In this paper, we use graph representation namely directed bipartite hypergraph to illustrate the relationship between overall data edits. Nodes in the graph not only contain variable information of data edits, but also the data edits itself. This makes the interaction between data edits can be seen as a basis to prevent new errors. We also introduce four parameters as determining the level of variables that are priorities for improvement. The goal is to minimize the number of variables must be fixed, but can eliminate all violations that occur. We evaluate the quality of the proposed structure by simulating data repairing. The results show that 100% of the data has decreased violations. 84% of them can be repaired to zero violations.