Abstract
String similarity join is an essential operation of data quality management and a key step to find the value of data. An incremental processing framework for string similarity join is proposed in this paper. Compared with the batching processing model, it can avoid the heavy time cost and the space cost brought by the duplicate similarity computation among historical strings and is suitable for processing data streams. We implement two algorithms: Inc-Join and Inp-Join. Inc-Join runs on a stand-alone machine while Inp-Join runs on a cluster with Spark environment. The experimental results show that this incremental processing framework can reduce the amount of string matching without affecting the join accuracy. When the data quantity becomes large, Inp-Join can make full use of the advantage of parallel processing and obtain a better performance than Inc-Join.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: International Journal of Computational Science and Engineering
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
