End-to-end Task Based Parallelization for Entity Resolution on Dynamic Data

Abstract

Entity resolution (ER) is the problem of finding which digital representations of entities correspond to the same real-world entity. In many Big Data scenarios, in addition to the problems of volume and variety that are commonly addressed in ER, data is continuously generated, which requires novel solutions to address the velocity problem.This paper presents a framework for end-to-end ER that incrementally and efficiently produces results as heterogeneous data streams in. These characteristics are achieved by proposing a novel functional model for ER on incremental or streaming data, and adopting task-based parallelization. Our evaluation demonstrates that even without parallelization, our framework outperforms state-of-the-art (batch) ER in terms of runtime and quality. We also validate that it can achieve high throughput and low latency on streaming data, paving the way to real-time ER.