Missing values compensation in duplicates detection using hot deck method

Abdulrazzak Ali,Siti A Asmai,Nurul A Emran

doi:10.1186/s40537-021-00502-1

Abdulrazzak Ali, Siti A Asmai + Show 1 more

Open Access

https://doi.org/10.1186/s40537-021-00502-1

Copy DOI

Abstract

Duplicate record is a common problem within data sets especially in huge volume databases. The accuracy of duplicate detection determines the efficiency of duplicate removal process. However, duplicate detection has become more challenging due to the presence of missing values within the records where during the clustering and matching process, missing values can cause records deemed similar to be inserted into the wrong group, hence, leading to undetected duplicates. In this paper, duplicate detection improvement was proposed despite the presence of missing values within a data set through Duplicate Detection within the Incomplete Data set (DDID) method. The missing values were hypothetically added to the key attributes of three data sets under study, using an arbitrary pattern to simulate both complete and incomplete data sets. The results were analyzed, then, the performance of duplicate detection was evaluated by using the Hot Deck method to compensate for the missing values in the key attributes. It was hypothesized that by using Hot Deck, duplicate detection performance would be improved. Furthermore, the DDID performance was compared to an early duplicate detection method namely DuDe, in terms of its accuracy and speed. The findings yielded that even though the data sets were incomplete, DDID was able to offer a better accuracy and faster duplicate detection as compared to DuDe. The results of this study offer insights into constraints of duplicate detection within incomplete data sets.

Highlights

Duplicate record is a common problem within data sets especially in huge volume databases
Both methods showed the same number of declared duplicates for the Restaurant data set, for the bigger data sets which are CD, MusicBrainz (A), and MusicBrainz(B), Duplicate Detection Project (DuDe) declared more duplicates than Duplicate Detection within the Incomplete Data set (DDID)
DDID showed a better performance than DuDe by producing fewer errors

Summary

Introduction

Duplicate record is a common problem within data sets especially in huge volume databases. Duplicate detection improvement was proposed despite the presence of missing values within a data set through Duplicate Detection within the Incomplete Data set (DDID) method. The results were analyzed, the performance of duplicate detection was evaluated by using the Hot Deck method to compensate for the missing values in the key attributes. Duplicates are detected by searching for all objects (or records) that represent the same real-world entity to ensure database consistency [6]. The similarity measure is based on equivalence relation where, suppose the real-world entities are er, es, and et, if er ≡ es and er ≡ et es ≡ et. The pairwise comparison approach relies on the use of similarity in attribute values to indicate real-world equivalence between the corresponding database entities. Some applications consider that false positives are worse than false-negative which results in their effect on the accuracy of the detected duplicates, whereas the opposite is true with other applications [8]

Methods

Results

Conclusion