Positive and negative association rule mining in Hadoop\u2019s MapReduce environment

Sikha Bagui,Probal Chandra Dhar

doi:10.1186/s40537-019-0238-8

Sikha Bagui, Probal Chandra Dhar

Open Access

https://doi.org/10.1186/s40537-019-0238-8

Copy DOI

Journal: Journal of Big Data	Publication Date: Aug 22, 2019
Citations: 17	License type: open-access

Affiliation: University of West Florida

Abstract

In this paper, we present a Hadoop implementation of the Apriori algorithm. Using Hadoop’s distributed and parallel MapReduce environment, we present an architecture to mine positive as well as negative association rules in big data using frequent itemset mining and the Apriori algorithm. We also analyze and present the results of a few optimization parameters in Hadoop’s MapReduce environment as it relates to this algorithm. The results are presented based on the number of rules generated as well as the run-time efficiency. We find that, a higher amount of parallelization, which means larger block sizes, will increase the run-time efficiency of the Hadoop implementation of the Apriori algorithm.

Highlights

Association rule mining, originally developed by [3], is a well-known data mining technique used to find associations between items or itemsets
In this paper we present an architecture for positive as well as negative association rule mining in the big data environment using Hadoop’s MapReduce environment using frequent itemset mining
Given the fact that repeated scans of the dataset are needed in the Apriori algorithm, the parallel and distributed structure of Hadoop should be availed of in an optimized way for mining positive as well as negative association rules in big data using the Apriori algorithm

Summary

Introduction

Association rule mining, originally developed by [3], is a well-known data mining technique used to find associations between items or itemsets. In this paper we present an architecture for positive as well as negative association rule mining in the big data environment using Hadoop’s MapReduce environment using frequent itemset mining. Given the fact that repeated scans of the dataset are needed in the Apriori algorithm, the parallel and distributed structure of Hadoop should be availed of in an optimized way for mining positive as well as negative association rules in big data using the Apriori algorithm.

Results

Conclusion