Swarm intelligent based online feature selection (OFS) and weighted entropy frequent pattern mining (WEFPM) algorithm for big data analysis

Abstract

During the past two decades, frequent pattern mining (FPM) has acquired the interests of many researchers: which involves extracting the itemsets from transactions, sequences from big dataset, which occurs frequently and to recognize from the molecular structures, the common sub graph. In this big data era, the unpredictable flow and huge quantity of data brings new challenges in FPM such as space and time complexity. In general, most of the research work focus on recognizing the patterns that occurs frequently, from the set of specific data, where the patterns within every transaction were definitely known a priori. Among these, the users focus only on the small part of this FP. In order to tackle such problems in the current scenario, it is necessary sometimes to select the important features alone, using appropriate FPM algorithms, in order to reduce the complexity level. The major objective of this work is to improve FPM mining results and improve classification accuracy of big dataset samples. To tackle the first challenge, the levy flight bat algorithm (LFBA) along with online feature selection (OFS) approach is proposed, which is used to filter the low quality features from the big data in an online manner. Subsequently to address the second challenge, a weighted entropy frequent pattern mining (WEFPM) is enforced for FPM, to accomplish better computation time when compared with other methods such as direct discriminative pattern mining (DDPMine) and iterative sampling based frequent itemset mining (ISbFIM), where enumeration of entire feature combinations were completed. So the WEFPM algorithm employed in this paper, targets to identify only the specific frequent patterns which are required by the user. By iterating this procedure, it assures that the acquired frequent patterns can be enumerated by using both the theoretical and empirical research, so that enumeration doesn’t proceed into a combinatorial explosion. And also, using the above said LFBA–OFS approach and WEFPM algorithm, frequent patterns that are different in nature, are generated for building high quality learning model. For finding the frequent patterns, here the minimum support threshold is matched with entropy. As a final step, multiple Kernel learning support vector machine is employed as a classifier, to evaluate the performance of the big data samples for efficiency and accuracy. Empirical study reveal that considerable progress is obtained in terms of accuracy and computation time when applied to UCI benchmark big datasets, using the proposed approach for efficient and effective FPM of the online features. It is clear that WEFPM is the most efficient method, because it produces higher average accuracy results of 92.34, 93.218, 91.374 and 87.87% values for adult, chess, hybo and sick dataset respectively. It outperforms when compared to other methods such as DDPMine and ISbFIM using an LIBSVM classifier.