Research on data cleaning method based on SNM algorithm

Abstract

With the rapid development of industry, every day there are huge amounts of industrial data in this process, there will be part of the data is duplicated, and it not only reduces the data quality in a certain extent, but also affects the enterprise to make a right decision, thereby reducing The productivity. In order to improve the quality of data, it's particularly important to clean the similar duplicate records. However, when SNM algorithm is used to detect similar records, we need to compare all the records in the window, and the time efficiency and accuracy are not high. Aiming at this defect, an improved dynamic fault-tolerant algorithm based on effective weight is proposed in this paper. Firstly, in the window according to the proportion of the length of two records will not be a duplicate record data excluded, reduce the times of comparing records, so as to improve the detection efficiency; Secondly, by setting the validity of the property factor, for the detection process Due to the miscarriages caused by missing attributes, a dynamic fault tolerance algorithm is proposed, which not only improves the efficiency of checking the weight but also ensures the accuracy of similar duplicate records detection. Finally, the experimental results show that, under the same experimental environment, the improved algorithm has obvious advantages both in terms of time efficiency and accuracy. Finally, the experimental results show that, in the same experimental environment, the improved algorithm has obvious advantages both in time efficiency and accuracy.