Map-optimize-reduce: CAN tree assisted FP-growth algorithm for clusters based FP mining on Hadoop

Abstract

Over the past era, Frequent Pattern Mining (FPM) is emerging as a significant approach to discover fascinating knowledge concealed in the data. However, preceding works failed to address the validation of FPM with user queries and also achieving better scalability and execution time is still bottleneck owing to difficulties in handling large dataset. To address this downside, our proposed work establishes FPM using extend version of MapReduce framework in Hadoop environment. Our proposed work comprises of five processes that are: 1) Preprocessing 2) Affinity Propagation (AP) based Clustering 3) Load Balancing 4) Map-Optimize-Reduce 5) Mining User Queries. Primarily, our proposed work performs preprocessing to remove data redundancy. To speed up the MapReduce framework, we propose AP clustering which generates effective clusters from the given dataset. Load balancing is executed to balance load among different blocks concerning where reputation is computed. To avoid oversight in scanning and minimal searching space in MapReduce, optimizer is included between Mapper and Reducer where Emperor Penguin Colony (EPC) optimization is used. Frequent patterns are mined using CANonical order (CAN) tree based Frequent Pattern (FP) growth which reduces execution time and frequent tree construction. User provides Mining_Request to the Hadoop and frequent patterns are mined for given query which is send back to the user. If user given query is not present in the CAN tree, then it sends Relevance Feedback as a recommendation to the user. Finally, we validate our proposed work performance with the previous works for succeeding metrics that are Execution Time, Response Time, Load Balancing Rate, and Scalability.