Abstract
Detecting approximate duplicate records in database is a key problem related to data quality. Given two lists of records, the duplicate detection problem consists of determining all pairs that are similar to each other, where the overall similarity between two records is defined based on domain-specific similarities over individual attributes constituting the record. In this paper, we present a synthetic approach for recognizing clusters of approximate duplicate records of multi-language data. The key ideas are: (1) an efficient algorithm for pre-processing multi-language data consists of Chinese words segmentation and Chinese named entity recognition; (2) an efficient pair-wise comparison method based on domain- specific similarities, especially, the string kernel method; (3) using a priority queue of duplicate clusters and representative records strategy to respond adaptively to the data scale.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
