Improving Data Processing Speed on Large Datasets in a Hadoop Multi-node Cluster using Enhanced Apriori Algorithm

Abstract

For large data, data mining methods were used on a Hadoop-based distributed infrastructure, using map reduction paradigm approaches for rapid data processing. Though data mining approaches are established methodologies, the Apriori algorithm provides a specific strategy for increasing data processing performance in big data analytics by applying map reduction. Apriori property is used to increase the efficiency of level-wise creation of frequent itemsets by minimizing the search area. A frequent itemset’s subsets must also be frequent (Apriori property). If an itemset is rarely, then all of its supersets are infrequent as well. We refined the apriori approach by varying the degree of order in locating frequent item sets in large clusters using map reduction programming. Fixed Pass Combined Counting (FPC) and Dynamic Pass Combined Counting (DPC) is a classical algorithm which are used for data processing from the huge datasets but their accuracy is not up to the mark. In this article, updated Apriori algorithms such as multiplied-fixed-pass combined counting (MFPC) and average time-based dynamic combined counting (ATDFC) are used to successfully achieve data processing speed. The proposed approaches are based on traditional Apriori core notions in data mining and will be used in the map-reduce multi-pass phase by ignoring pruning in some passes. The optimized-MFPC and optimized-ATDFC map-reduce framework model algorithms were also presented. The results of the experiments reveal that MFPC and ATDFC are more efficient in terms of execution time than previously outmoded approaches such as Fixed Pass Combined Counting (FPC) and Dynamic Pass Combined Counting (DPC). In a Hadoop multi-node cluster, this paradigm accelerates data processing on big data sets. Previous techniques were stated in terms of reducing execution time by 60–80% through the use of several passes. Because of the omitted trimming operation in data pre-processing, our proposed new approaches will save up to 84–90% of that time.