Research on Classification and Redundant Information Filtering of Massive Data in Big Database

Abstract

The precise classification of massive information in big database was researched in this paper, and also the redundant information as the interference should be filtered in the subject. According to the traditional data classification method, the frequency points were concentrated and the data classification frequent points were not easy to be eliminated. The nodes classification technology with low self-adaptive property refused the nodes in high disturbance and in the deep attenuation parts, and then the classification precision and the immunity of the disturbance property were limited greatly. A new optimum data classification method and the redundant information model were proposed based on the chaotic probability analysis. The classification error rates was mapped as a probability density function based on the channel mapping function method, the classification probability was allocated with this probability density function. The random series which could reflect the essential feature was produced based on the chaotic probability analysis method which could meet to the demands of the random frequency classification. And the data clustering and optimization classification was realized finally. Simulation was taken with the KDD_CUP2009 experimental big database, and simulation result shows that the proposed method can classify each type of the data effectively. The performance of the data classification is perfect, comparing to the traditional neural net fuzzy c-means method, the classification precision rate was improved by 17.8% It show that the model and algorithm has excellent classification performance and can be taken in the application such as data mining, fault diagnosis and target recognition as engineering practice.