Resolving Conflicts in Data Through Curation-informed Weight Distribution Networks

Abstract

Missing and conflicting data values create problems when integrating datasets from multiple collections. Moreover, when the collections to be integrated are large and continuously updated, it is not feasible to manually resolve these problems. Instead, disagreements and gaps should be resolved in an automated fashion. To achieve good quality integrated datasets automatically we introduce the Curation-informed Weight Distribution Network (CiWDN), a method that suggests which collection is more reliable in providing a data value in question. CiWDN adapts the PageRank algorithm (PR) to assign and distribute weights across data fields present in the different collections. Weight assignment is rooted in data curation best practices as metrics of a collection's reliability. The metrics include: a) data completeness, b) data coincidence, and c) data consistency over time. Final weights used as collection ranks provide the basis to resolve conflicts between different collections contributing a data value for a given field. CiWDN relies on a data dictionary that normalizes fields across collections, and is implemented on a graph database. We demonstrate CiWDN’s capability using the case of ASTRIAGraph, a knowledge system built to increase transparency of activities in Earth’s orbital environment. CiWDN can assess the reliability of data collections that conflict on space object characteristic data fields, which can be used to resolve the differences. This method for computing collections' reliability can be ported to curate other types of large integrated datasets for use in machine learning and other data-driven applications.