Abstract
Online data deduplication aims to identify records that represent the same purpose on a continuous data flow environment. It must be able to process a range of information with high effectiveness and no delays. The purpose of this paper is to introduce a developed tool entitled Redblock, for real-time data deduplication, using a distributed platform for online processing combined with an Inverted Index. During the experimental evaluation, Redblock managed to provide good preliminary results in terms of efficiency and effectiveness in a database.
Highlights
Data integration provides information in an easy and accessible way by consolidating different sources of data into a single repository
This paper proposes a new tool for online deduplication focusing on the blocking process
It is possible to verify that Redblock maintained a high quality
Summary
Data integration provides information in an easy and accessible way by consolidating different sources of data into a single repository. A fundamental task is identifying entities (i.e., records, documents or texts) that are already inserted in the database so it is not necessary for them to be allocated again. This process is known as data deduplication. Blocking corresponds to the process of generating pair candidates It means that all records must be analyzed in the search of potential duplicates. In order to understand our proposed approach, this section presents main methods involving data blocking and the Apache Storm platform which turns possible to develop applications that require massive data processing in real time
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
