Parallel Association Rule Mining Research Articles

Operating Optimization of Steam Turbine Unit Based on Big Data Parallel Association Rule Mining

Operating Optimization of Steam Turbine Unit Based on Big Data Parallel Association Rule Mining

Research on the innovation of the path of ideological and political education in universities based on data mining

Abstract This paper first describes the parallel association rule mining method for the big data implementation process. To perform similar item merging, each data item in the dataset is scanned, the dataset is divided, and then the similarity degree is calculated. And the dynamic support threshold is obtained based on the idea of genetic iteration of the genetic algorithm to obtain the optimal solution. In order to conduct data mining, the mining process is then transferred to the MapReduce computing platform while the data output from the Map step is parallel LZO compressed. Finally, in the age of big data, the ideological and political work in colleges is mined and analyzed using a data mining algorithm, and route innovation remedies are proposed. According to the current state of ideological and political education, 77.6% of respondents believe that the ideological and political management system is not ideal, and 61.48 percent believe that the ideological and political content is fragmented and that data-based information makes it challenging to supervise. The immaturity of new media development and the growth of technological challenges amounted to 46%. After creative approaches, ideological and political education teaching satisfaction has grown by 15-20 percentage points. Ideological and political education at universities under the impact of big data should grab the possibilities and overcome the difficulties to achieve sustainability and innovative development.

Parallel incremental association rule mining framework for public opinion analysis

Internet public opinion association rule mining (POARM) has garnered significant attention from a larger group of netizens. However, most POARM methods have been applied to post-event analysis, which has poor timeliness and a low efficiency. Therefore, the real-time monitoring of public opinion association rules is lacking. To address this problem, we propose the parallel Incremental POARM Framework (IPOARM), which improves the timeliness of association rule mining in two ways: 1) using an incremental merge method to consider both inserted and deleted public opinion transaction sets and reuse previous frequent itemsets to reduce redundant computation and 2) employing a parallel implementation of big data process platforms. Moreover, the flexible association rule mining (ARM) algorithm selection structure of IPOARM enables users to freely select suitable ARM algorithms. We represent four classic transaction sets as public opinion transaction sets and compare the IPOARM framework with two novel incremental association rules mining algorithms. Our evaluations indicate that the IPOARM framework can discover Internet public opinion association rules quickly, implying that it can be easily integrated into existing big data processing platforms and that it significantly improves the mining accuracy and efficiency by 12.756% and 29.371%, respectively.

Research on parallel association rule mining of big data based on an improved K-means clustering algorithm

Research on parallel association rule mining of big data based on an improved K-means clustering algorithm

Intelligent Construction and Mechanism of Educational Big Data Information for Resource Sharing

The word “sharing” is not unfamiliar to everyone now, but many people do not understand the sharing of educational resources (ER), and the distribution of ER does not seem to be the topic that people are most concerned about. This paper is aimed at conducting research on the construction of educational big data information intelligence based on resource sharing. A parallel association rule mining approach for educational big data is proposed in this research. Now, the educational big data is classified, and then, the data is operated. At the same time, a CcG-based educational information resource sharing platform has also been built for educational resource sharing, which can be used for cloud download and upload functions of cloud computing (CcG) for teaching and educational resource sharing. The experimental results of this paper show that the educational information resource sharing platform in this paper processes about 150 information per second, and its service performance must be better than that of traditional educational platforms.

CARs-Lands: An associative classifier for large-scale datasets

Associative classifiers are one of the most efficient classifiers for large datasets. However, they are unsuitable to be directly used in large-scale data problems. Associative classifiers discover frequent/rare rules or both in order to produce an efficient classifier. Discovery rules need to explore a large solution space in a well-organized manner; hence, learning of the associative classification methods of large datasets is not suitable on large-scale datasets because of memory and time-complexity constraints. The proposed method, CARs-Lands, presents an efficient distributed associative classifier. In CARs-Lands, first, a modified dataset is generated. This new dataset has sub-datasets that are completely appropriate to produce classification association rules (CARs) in a parallel manner. The produced dataset by CARs-Lands contains two types of instances: main instances and neighbor instances. Main instances can be either real instances of training dataset or meta-instances, which are not in the training dataset; each main instance has several neighbor instances from the training dataset, which together form a sub-dataset. These sub-datasets are used for parallel local association rule mining. In CARs-Lands, local association rules lead to more accurate prediction, because each test instance is classified by the association rules of their nearest neighbors in the training datasets. The proposed approach is evaluated in terms of accuracy on six real-world large-scale datasets against five recent and well-known methods. Experiment results show that the proposed classification method has high prediction accuracy and is highly competitive when compared to other classification methods.

PARALLEL AND DISTRIBUTED ASSOCIATION RULE MINING ALGORITHMS: A RECENT SURVEY

Data investigation is an essential key factor now a days due to rapidly growing electronic technology. It generates a large number of transactional data logs from a range of sources devices. Parallel and distributed computing is a useful approach for enhancing the data mining process. The aim of this research is to present a systematic review of parallel association rule mining (PARM) and distributed association rule mining (DARM) approaches. We have observed that the parallelized nature of Apriori, Equivalence class, Hadoop (MapReduce), and Spark proves to be very efficient in PARM and DARM environment. We conclude that this comprehensive review, references cited in this article will convey foremost hypothetical issues and a guideline to the researcher an interesting research direction. The most important hypothetical issue and challenges include the large size of databases, dimensionality of data, indexing schemes of data in the database, data skewness, database location, load balancing strategies, methods of adaptability in incremental databases and orientation of the database.

Academic research trend analysis based on big data technology

Big data technology can well support the analysis of academic research trends, which requires the ability to process an enormous amount of metadata efficiently. On this point, we propose an academic trend analysis method that exploits a popular topic model for paper feature extraction and an influence propagation model for field influence evaluation. We also propose a parallel association rule mining algorithm based on Spark to accelerate trend analysis process. The algorithm can take the advantages of Spark memory architecture to enhance the iterative speed of traditional algorithms. Experimentally, a vast amount of paper metadata was collected from four popular digital libraries: ACM, IEEE, Science Direct and Springer, serving as the raw data for our final feature dataset. Focusing on the hotspot of cloud computing, our result demonstrates that the most relevant topics to cloud computing have been changing these years from basic research to applied research, and from a microscopic point of view, the development of cloud computing related fields presents a certain periodicity.

An Efficient Parallel Association Rule Mining Algorithm based on Map Reduce Framework

An Efficient Parallel Association Rule Mining Algorithm based on Map Reduce Framework

An Efficient Parallel Association Rules Mining Algorithm for Fault Diagnosis

With the development of Internet industry, equipment data is increasing. The traditional method is not suitable for processing large data. Aiming at inefficient problem of Apriori algorithm when mining very large database, an efficient parallel association rules mining algorithm (Advanced Pruning Parallel Apriori Algorithm) based on a cluster is presented. APPAA algorithm can enhance the mining efficiency, as well as the system’s extension. Experimental results show that APPAA algorithm cuts down 85% mining time of Apriori, and it has good characteristics of parallel and expandable.so it is suitable for mining very large size database of fault diagnosis.

A sparse memory allocation data structure for sequential and parallel association rule mining

In this paper, we present a sparse memory allocation data structure for sequential and parallel data mining. We explored three algorithms utilizing the proposed data structure: MASP-tree, apriori-TID, and FP-growth. We modified the data structure of apriori-TID and FP-growth algorithms to reduce memory allocation cost. Five data sets are used for comparison. The results show that the modified apriori-TID has a higher speed-up than the modified FP-growth when the proposed data structure is used. A maximum speed-up of 3.42 is observed when MASP algorithm is tested.

Research on Parallel Association Rules Mining Algorithm Based on Hadoop

The purpose of association rules mining is to find rules which can meet the minimum support and minimum confidence from a large quantity of data. To find the valid association rules efficiently, we had a comprehensive analysis on some well-know parallel association rules mining algorithms and proposes a new parallel association rules mining algorithm (Array Based on Hadoop, short for ABH) based on the cloud computing platform. The ABH scans the database only once, uses the 0/1 array to represent one of the transactions and to record the frequency of the same transaction. Moreover, by utilizing the random access characteristics of the array and the special nature of the frequent itemset, the ABH can reduce the quantity of frequent candidate itemset effectively and find the frequent itemset quickly. We have compared the ABH with two classical algorithms CD and DD through experiment; we can find that ABH outperforms CD and DD.

A new parallel association rule mining algorithm on distributed shared memory system

Frequent itemset finding is the most time consuming step in analysing large transactional databases. The use of sequential algorithms cannot give analytical ability for such very large databases especially in terms of run-time performance. Therefore, we must rely on high performance parallel computing. In this paper, we present a new parallel algorithm for frequent itemset mining, called 'HorVertical' algorithm. This algorithm introduces a new database partitioning called 'HorVertical' partitioning. This technique in partitioning the database reduces the dependency in the parallel computation and gives new properties to reduce the computations. The algorithm passes the database only one time and starts a new stage with the finished itemsets while some other itemsets in the same stage have not been finished yet. We present the result on the performance of our algorithm on various databases, and compare it against well known algorithms.

Parallel Association Rule Mining on Heterogeneous System

Association Rule Mining from transaction–oriented databases is one of the important process that finds relation between items and plays important role in decision making. Parallel algorithms are required because of large size of the database to be mined. Most of the algorithms designed were for homogeneous system uses static load balancing technique which is far from reality. A parallel algorithm for heterogeneous system is regarded as one of the most promising platforms for association rule mining. In this paper we propose a simple parallel algorithm for association rule mining on heterogeneous system with dynamic load balancing based on Par-Maxclique algorithm. We maintain one linked list at the scheduler end that keep track of load value of every processor and each processor is having a job queue associated with it which is served in First come first basis. On the basis of load value scheduler directs the migration of task from heavy loaded to least loaded processor in the cluster during the execution and thus balances load dynamically in a cluster.

Parallel mining of association rules from text databases

In this paper, we propose a new algorithm named Parallel Multipass with Inverted Hashing and Pruning (PMIHP) for mining association rules between words in text databases. The characteristics of text databases are quite different from those of retail transaction databases, and existing mining algorithms cannot handle text databases efficiently because of the large number of itemsets (i.e., sets of words) that need to be counted. The new PMIHP algorithm is a parallel version of our Multipass with Inverted Hashing and Pruning (MIHP) algorithm (Holt, Chung in: Proc of the 14th IEEE int'l conf on tools with artificial intelligence, 2002, pp 49---56), which was shown to be quite efficient than other existing algorithms in the context of mining text databases. The PMIHP algorithm reduces the overhead of communication between miners running on different processors because they are mining local databases asynchronously and prune the global candidates by using the Inverted Hashing and Pruning technique. Compared with the well-known Count Distribution algorithm (Agrawal, Shafer in: (1996) IEEE Trans Knowl Data Eng 8(6):962---969), PMIHP demonstrates superior performance characteristics for mining association rules in large text databases, and when the minimum support level is low, its speedup is superlinear as the number of processors increases. These experiments were performed on a cluster of Linux workstations using a collection of Wall Street Journal articles.

Parallel mining and application of fuzzy association rules

Quantitative attributes are partitioned into several fuzzy sets by using fuzzy c-means algorithm. Fuzzy c-means algorithm can embody the actual distribution of the data, and fuzzy sets can soften the partition boundary. Then, we improve the search technology of apriori algorithm and present the algorithm for mining fuzzy association rules. As the database size becomes larger and larger, a better way is to mine fuzzy association rules in parallel. In the parallel mining algorithm, quantitative attributes are partitioned into several fuzzy sets by using parallel fuzzy c-means algorithm. Boolean parallel algorithm is improved to discover frequent fuzzy attribute set, and the fuzzy association rules with at least a minimum confidence are generated on all processors. The experiment results implemented on the distributed linked PC/workstation show that the parallel mining algorithm has fine scaleup, sizeup and speedup. Last, we discuss the application of fuzzy association rules in the classification. The example shows that the accuracy of classification systems of the fuzzy association rules is better than that of the two popular classification methods: C4.5 and CBA.

An Efficient Graph-Based Method for Parallel Mining Problems

Parallel association rules mining is a noticeable problem in data mining. However, little work has been proposed to deal with three important issues: (1) less memory usage; (2) less communication, among the involved computers, over the network; and (3) load balance among computers. In this paper, we present a graph-based scheme to solve the parallel mining problem by applying independent groups (clusters of maximal cliques). To bring the three issues to a close, the purpose of the independent groups aims at dividing a database into several independent sub-databases, so each sub-database can be employed independently to perform mining algorithms. To emphasis the effectiveness of the graph-based scheme, we adopt the independent groups not only for maximal large itemsets mining but also for general large itemsets mining. The experimental results show that our scheme can improve the efficiency for parallel mining when the independent groups are well-organized and designed.

Parallel mining of association rules

We consider the problem of mining association rules on a shared nothing multiprocessor. We present three algorithms that explore a spectrum of trade-offs between computation, communication, memory usage, synchronization, and the use of problem specific information. The best algorithm exhibits near perfect scaleup behavior, yet requires only minimal overhead compared to the current best serial algorithm.