Big Data Outlier Detection Algorithm Based on Grid

Abstract

To study the outlier detection algorithm of big data in grids, the outliers were cleaned by the grid-based LOF algorithm under the Hadoop platform. The algorithm of grid pruning and LOF outlier detection was introduced. For the inaccuracy of mesh pruning, the concept of clustering radius was proposed. By introducing clustering radius, the detection accuracy of grid pruning was greatly improved. At the same time, the concept of grid numbering was proposed for the case that LOF algorithm and MapReduce could not be perfectly combined. Through the mesh number, the out-of-line detection of LOF was performed using the parallelism mechanism of MapReduce. Finally, the implementation of grid-based LOF algorithm on MapReduce was introduced. The results showed that the computational complexity was greatly reduced and the detection efficiency was improved. In summary, it provides a certain reference for the research on EMU big data cleaning.