Entity Resolution Algorithm for Heterogeneous Data Sources

Abstract

Entity resolution is an important step in data cleaning and data integration, which can identify records from different data sources that refer to the same entity. For the problem of information loss faced by schema-agnostic entity resolution in heterogeneous environments, the problem of schema heterogeneity is studied and a schema-matched heterogeneous entity resolution algorithm S-HER is proposed. The authors introduce the concept of synonyms, use semantic information to match attribute names and cluster attribute values according to data type and distribution. A schema matching method based on semantics and clustering is proposed. The matching scores of attribute names and attribute values are jointly learned to combine schema information and data instances. After schema matching, a complete-weight bipartite graph optimal matching algorithm is introduced to calculate the similarity of record pairs, and weights are assigned to different attributes according to the contribution of attribute differences. The calculation of the similarity of the record pairs is optimized. Experiments on real datasets show that the proposed method is better than the existing entity resolution methods on accuracy and efficiency.