Duplicate record detection approach based on sentence embeddings

Abstract

Duplicate record detection is a crucial task for data cleaning. Records representation is among the main challenges of this task. Word embeddings models have been widely applied in an attempt to improve records representation. However, despite the improvements made by word embeddings to enhance the semantic aspect, duplicate record detection results is still insufficient In this paper, we present a duplicate record detection approach based on sentence embeddings, where each attribute is viewed as a sentence. First, universal sentence encoder model is used to embed the values of records’ attributes into embeddings vectors. Afterwards, based on the created vectors, similarity vectors between the record pairs are computed. Finally, support vector machine algorithm is used to classify the similarity vectors. Experiments on two datasets (Cora and Restaurant) show that our proposal outperforms state-of-the-art baselines and leads to significant improvements in duplicate record detection effectiveness.