Large scale data mining based on data partitioning

Abstract

Dealing with very large databases is one of the defining challenges in data mining research and development. Some databases are simply too large (e.g., with terabytes of data) to be processed at one time. For efficiency and space reasons, partitioning them into subsets for processing is necessary. However, since the number of itemsets in each partitioned data subset can be a combinatorial amount and each of them may be a large itemset in the original database, data mining results from these subsets can be very large in size. Therefore, the key to data partitioning is how to aggregate the results from these subsets. It is not realistic to keep all results from each subset, because the rules from one subset need to be verified for usefulness in other subsets. This article presents a model of aggregating association rules from different data subsets by weighting. In particular, the aggregation efficiency is enhanced by rule selection.