Distributed Data Mining Algorithms Research Articles

Research on gaussian mixture model and its distributed data mining algorithm in wireless sensor networks

Optimization of the routing represents an important challenge when considering the rapid development of Wireless Sensor Networks (WSN), which involve efficient energy methods. Applying the effectiveness of a Deep Neural Network (DNN) and a Gaussian Mixture Model (GMM), the present article proposes an innovative method for attaining Energy-Efficient Routing (EER) in WSN. When it comes to dealing with dynamic network issues, conventional routing protocols generally conflict, resulting in unsustainable Energy consumption (EC). By applying algorithms based on data mining to adapt routing selections in an effective procedure, the GMM + DNN methodology that has been developed is able to successfully address this problem. The GMM is a fundamental Feature Extraction (FE) method for accurately representing the features of statistical analysis associated with network parameters like signal frequency, the amount of traffic, and channel states. By learning from previous data collection, the DNN, which relies on these FE, provides improved routing selections, resulting in more efficient use of energy. Since routing paths are constantly optimized to ensure dynamic adaptation, where less energy is used, networks last longer and perform more efficiently. Network simulations highlight the GMM + DNN method’s effectiveness and depict how it outperforms conventional routing methods while preserving network connectivity and data throughput. The GMM + DNN’s adaptability to multiple network topologies and traffic patterns and its durability make it an efficient EER technique in the diverse WSN context. The GMM + DNN achieves an EC of 0.561 J, outperforming the existing state-of-the-art techniques.

Development Trend of Intelligent Transportation in the Era of Big Data

The arrival of the big data era has brought great convenience to people’s real life. All aspects of our lives follow the development of the big data era. In the context of the era of big data, through data collection and analysis, the current urban transportation has been reasonably developed, laying a good foundation for the development of intelligent transportation in the future, and can solve various current problems. Based on the analysis of the traffic system, this article finds that the real-time road condition prediction of common functions is highly correlated with route guidance. Therefore, this paper uses a method based on historical data to predict road conditions, and uses the prediction results to dynamically allocate road segments, and then uses the relevant single-source shortest path algorithm to simulate the optimal path. By analyzing the application advantages of data mining in intelligent transportation, Baum Welch’s algorithm is used for data mining in a real-time traffic prediction system based on historical data. Real-time road conditions are matched with historical data models to predict road conditions. Finally, analyzed and designed the big data cloud platform framework applied to it to achieve higher efficiency. The experimental research results show that, by studying the development of urban intelligent transportation systems, the purpose of this paper is to make better use of existing resources to process data more efficiently, and then combine distributed data mining algorithms to optimize the processed data and apply it to in the intelligent transportation system, an intelligent management and control system that is more efficient, accurate, and user experience better can be established.

Research on human performance evaluation model based on neural network and data mining algorithm

In order to effectively evaluate personnel performance, a distributed data mining algorithm for spatial networks based on BP neural wireless network is proposed. In the cloud computing environment, an excavator is used to construct multiple input multiple output spatial network data, analyze the data structure, and perform redundant data compression of massive data through time-frequency feature extraction. Combined with the adaptive matching filtering method, the characteristics of the data are matched. The spatial frequency feature extraction method is used to locate the features of the multiple-input multiple-output spatial network data. In order to improve the accuracy of data mining, the BP neural network is used to classify and identify the extracted data features to achieve the optimization of data mining. A wireless sensor network is a wireless network composed of a large number of stationary or moving sensors in a self-organizing and multi-hop manner. It cooperatively senses, collects, processes, and transmits the information of the perceived objects in the geographical area covered by the network and finally puts these The information is sent to the owner of the network. This algorithm improves the accuracy of personnel performance evaluation, simultaneously establishes a hierarchical analysis and quantitative evaluation model for the performance of government managers, and adjusts the results of hierarchical statistical analysis on government administrators as needed. The performance evaluation and optimization of government administrators were introduced. The empirical analysis results show that the method has higher accuracy for government managers’ performance evaluation, higher efficiency of big data processing, and better integration.

DENCAST: distributed density-based clustering for multi-target regression

Recent developments in sensor networks and mobile computing led to a huge increase in data generated that need to be processed and analyzed efficiently. In this context, many distributed data mining algorithms have recently been proposed. Following this line of research, we propose the DENCAST system, a novel distributed algorithm implemented in Apache Spark, which performs density-based clustering and exploits the identified clusters to solve both single- and multi-target regression tasks (and thus, solves complex tasks such as time series prediction). Contrary to existing distributed methods, DENCAST does not require a final merging step (usually performed on a single machine) and is able to handle large-scale, high-dimensional data by taking advantage of locality sensitive hashing. Experiments show that DENCAST performs clustering more efficiently than a state-of-the-art distributed clustering algorithm, especially when the number of objects increases significantly. The quality of the extracted clusters is confirmed by the predictive capabilities of DENCAST on several datasets: It is able to significantly outperform (p-value <0.05) state-of-the-art distributed regression methods, in both single and multi-target settings.

A system based on Hadoop for radar data analysis

Nowadays, radar technique is widely used in many applications, such as electronic warfare, weather prediction, navigation, and self-driving car. Large amounts of radar data has been generated by the wide use of radar technique. Analyzing radar data has a quite important role in daily life, as well as in military. Finding the frequent sequences in radar data is significant for radar data analysis. However, traditional analysis systems using standalones cannot process big data due to the four features of big data, namely, volume, velocity, variety and value. Many distributed frameworks are promising for processing large scale data sets, such as Hadoop and Spark. Therefore, to deal with the problem of finding frequent sequences from large amounts of radar data, we built a system based on Hadoop and Spark. With the combination of Hadoop and Spark, we can store big data, as well as analyze big radar data more easily. In the proposed system, Hadoop distributed file system offers stable data storage, and Spark offers efficient in-memory calculation. In this paper, a three-node Hadoop–Spark cluster was built to perform the distributed data mining algorithm. Additionally, to make the analysis of radar data accurate, we proposed ideas of preprocessing radar data and post processing mining results. Experimental results show that the system we proposed can analyze the large amounts of radar data efficiently and accurately.

Privacy-preserving of SVM over vertically partitioned with imputing missing data

Most distributed data mining algorithms can efficiently manage and mine complete data from distributed resources. However, for an incomplete data some modifications are required in order to perform distributed data mining techniques and maintaining the privacy of the sensitive information to provide pretty good results of data mining. Classification is important tasks of data mining aimed at discovering knowledge and classify new instances. SVM is classified as one of the most important algorithm used for classification problems in several various spheres. In this paper, we proposed a new distributed privacy-preserving protocol with multiple imputations of missing or incomplete data. More so, a multiple imputations based on multivariate imputation by chained equations is used for missing data and Paillier cryptosystem for maintaining the privacy of the participants. Finally we constructed a global SVM model by introducing a third party (semi-honest approach) over vertical partition data based in Gram matrix without revealing the privacy of the data and used to classify new instances. The performance evolution of the proposed protocol was investigated while using accuracy metric on the distributed and centralized data. Results of our experiments reveal that the accuracy is the same as centralized data and achieve better results with imputed data while compared with omitted data. The performance of distributed data on our protocol achieves better processing time compared with centralized data.

Research and development of intelligent cutting database cloud platform system

In order to promote the integrated application of the Internet of Things and cloud platform computing technology in intelligent manufacturing system, a cloud platform of cutting database was established in this paper. With the help of multi-sensor and distributed storage technology, mathematics model and data mining algorithm, functions of fast accessing and storing cutting data, intelligent modeling and analysis, efficient mining and optimizing processing parameters can be realized in the system, solving the key problems of analyzing cutting force and controlling surface quality in turning difficult-to-cut materials. Data equipment and users in the system were connected as a whole through wireless network, which can provide intelligent service of data querying, modeling, and optimizing through the cloud server. The study made up for the drawbacks in current intelligent database system such as oversimplified platform architecture, low level of intelligence, and low utilization of historical data, providing an efficient method of managing and applying big data in cutting process for the production system.

Recommendation System Based On Association Rules For Distributed E-Learning Management Systems

Abstract Traditional Learning Management Systems are installed on a single server where learning materials and user data are kept. To increase its performance, the Learning Management System can be installed on multiple servers; learning materials and user data could be distributed across these servers obtaining a Distributed Learning Management System. In this paper is proposed the prototype of a recommendation system based on association rules for Distributed Learning Management System. Information from LMS databases is analyzed using distributed data mining algorithms in order to extract the association rules. Then the extracted rules are used as inference rules to provide personalized recommendations. The quality of provided recommendations is improved because the rules used to make the inferences are more accurate, since these rules aggregate knowledge from all e-Learning systems included in Distributed Learning Management System.

Decentralized detection of network attacks through P2P data clustering of SNMP data

The goal of Network Intrusion Detection Systems (NIDSs) is to protect against attacks by inspecting network traffic packets, for instance, looking for anomalies and signatures of known attacks. This paper illustrates an approach to attack detection that analyzes just the standard statistics automatically generated by the Simple Network Management Protocol (SNMP) through unsupervised distributed data mining algorithms. We describe the design of a decentralized system composed of a peer-to-peer network of monitoring stations: each of them continuously gathers SNMP statistical observations about the network traffic and runs a distributed data clustering algorithm in cooperation with other stations. This progressively leads to the construction of a traffic model capable to detect undergoing attacks on later observations, including potentially previously unknown attacks. To estimate the accuracy of the described system, we performed an extensive number of distributed data clustering processing on data sets of SNMP observations generated from real traffic.

Research on Association Rule Mining Algorithm Based on Distributed Data

Existing data miming algorithms have mostly implemented data mining under centralized environment, but the large-scale database exists in the distributed form. According to the existing problem of the distributed data mining algorithm FDM and its improved algorithms, which exist the problem that the frequent itemsets are lost and network communication cost too much. This paper proposes a association rule mining algorithm based on distributed data (ARADD). The mapping marks the array mechanism is included in the ARADD algorithm, which can not only keep the integrity of the frequent itemsets, but also reduces the cost of network communication. The efficiency of algorithm is proved in the experiment.

A Distributed Association Rules Mining Algorithm

Data mining has attracted a great deal of attention in the information industry in recent years and can be used for applications rangning from business management, production control, and science exploration etc. Most of the existing data mining algorithms are processing in the centralized systems; however, at present large database is usually distributed. Compared with the frequent itemsets lost and high communication traffic in distributed database conventional and improved algorithm FDM, An improved distributed data mining algorithm LTDM based on association roles is proposed. LTDM algorithm introduces the mapping indicated array mechanism to keep the integrity of frequent itemsets and decrease the communication traffic. The experimental results prove the efficiency of the proposed algorithm. The algorithm can be applied to information retrieval and so on in the digital library.

Peer‐to‐peer distributed text classifier learning in PADMINI

AbstractPopular Internet document repositories, such as online newspapers, digital libraries, and blogs store large amount of text and image data that are frequently accessed by large number of users. Users' input through collaborative commenting or tagging can be very useful in organizing and classifying documents. Some web sites (e.g. Google Image Labeler) support a collection of tags and labels, but a large fraction of these sites do not currently support such activities. Moreover, relying upon centrally controlled web‐service providers for such support is probably not a good idea if the objective is to make the collaborative inputs publicly available. Often, business entities offering such web‐based tagging environments end up owning and monetizing the result of the collective effort. This paper takes a step toward addressing this problem—it proposes a peer‐to‐peer (P2P) system (PADMINI), powered by distributed data mining algorithms. In particular, it focuses on learning a P2P classifier from tagged text data. This paper describes the PADMINI system and the distributed text classifier learning components; text classification is posed as a linear program and an asynchronous distributed algorithm is used to solve it. It also presents extensive empirical results on text data obtained from the Hubble Space Telescope (HST) proposal abstract database. Copyright © 2012 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 2012

Performance Analysis of Distributed Association Rule Mining with Apriori Algorithm

One of the most crucial problem in data mining is association rule mining. It requires large computation and I/O traffic capacity. One approach to resolve this problem is the use of distributed data mining algorithms in grid. It offers an effective way to mine for large data sets. Therefore, we implemented distributed data mining with Apriori algorithm in grid environment. However, usage of grid environment raises some issues about the optimization of the Apriori algorithm, especially the cost of the node to node communication and data distribution. In this paper, an Optimized Distributed Association rule mining approach for geographically distributed data is introduced in parallel and distributed environment; therefore, it reduces communication costs.

An ensemble-based evolutionary framework for coping with distributed intrusion detection

A distributed data mining algorithm to improve the detection accuracy when classifying malicious or unauthorized network activity is presented. The algorithm is based on genetic programming (GP) extended with the ensemble paradigm. GP ensemble is particularly suitable for distributed intrusion detection because it allows to build a network profile by combining different classifiers that together provide complementary information. The main novelty of the algorithm is that data is distributed across multiple autonomous sites and the learner component acquires useful knowledge from this data in a cooperative way. The network profile is then used to predict abnormal behavior. Experiments on the KDD Cup 1999 Data show the capability of genetic programming in successfully dealing with the problem of intrusion detection on distributed data.

Approximate Distributed K-Means Clustering over a Peer-to-Peer Network

Data intensive peer-to-peer (P2P) networks are finding increasing number of applications. Data mining in such P2P environments is a natural extension. However, common monolithic data mining architectures do not fit well in such environments since they typically require centralizing the distributed data which is usually not practical in a large P2P network. Distributed data mining algorithms that avoid large-scale synchronization or data centralization offer an alternate choice. This paper considers the distributed K-means clustering problem where the data and computing resources are distributed over a large P2P network. It offers two algorithms which produce an approximation of the result produced by the standard centralized K-means clustering algorithm. The first is designed to operate in a dynamic P2P network that can produce clusterings by ldquolocalrdquo synchronization only. The second algorithm uses uniformly sampled peers and provides analytical guarantees regarding the accuracy of clustering on a P2P network. Empirical results show that both the algorithms demonstrate good performance compared to their centralized counterparts at the modest communication cost.

A grid portal for solving geoscience problems using distributed knowledge discovery services

This paper describes our research effort to employ Grid technologies to enable the development of geoscience applications by integrating workflow technologies with data mining resources and a portal framework in unique work environment called MOSÈ. Using MOSÈ, a user can easily compose and execute geo-workflows for analyzing and managing natural disasters such as landslides, earthquakes, floods, wildfires, etc. MOSÈ is designed to be applicable both for the implementation of response strategies when emergencies occur and for disaster prevention. It takes advantage of the standardized resource access and workflow support for loosely coupled software components provided by web/grid services technologies. The integration of workflows with data mining services significantly improves data analysis. Geospatial data management and mining are critical areas of modern-day geosciences research. An important challenge for geospatial information mining is the distributed nature of the data. MOSÈ provides knowledge discovery services based on the WEKA data mining library and novel distributed data mining algorithms for spatial data analysis. A P2P bio-inspired algorithm for distributed spatial clustering as an example of distributed knowledge discovery service for intensive data analysis is presented. A real case application for the analysis of landslide hazard areas in the Campania Region near the Sarno area shows the advantages of using the portal.

Incremental DataGrid Mining Algorithm for Mobility Prediction of Mobile Users

Problem statement: Mobility prediction is the important issue in Personal Communication Systems (PCS). Mobile users moving logs are stored in data grid located in different locations. Distributed data mining algorithm is applied on this moving logs to generate the mobility pattern of mobile users. As new moving logs are added to the data grid, existing mobility pattern becomes invalid and it should be updated. One of the existing work to derive the new mobility pattern is re-executing the algorithm from scratch results in excessive computation. Approach: We had designed new incremental algorithm by maintaining infrequent mobility patterns, which avoids unnecessary scan of full database. Incremental data mining algorithm taken lesser time to compute new mobility patterns. The discovered location patterns can be used to provide various location based services to the mobile user by the application server in mobile computing environment. Data grid provided geographically distributed database for computational grid which implements incremental data mining algorithm. We built data grid system on a cluster of workstation using open source globus toolkit 4.0 and Message Passing Interface extended with Grid Services (MPICH-G2). Results: The experiments were conducted on original data sets and data were added incrementally and the computation time was recorded for each data sets. The performance improvement for increment size of 100 K was about 55% for 0.20% support count and it is increased to 60% for 0.25% support count. The performance is increased about 65% for the support count 0.30%. Conclusion: We analyzed our results with various sizes of data sets and the proof shows the time taken to generate mobility pattern by incremental mining algorithm is less than re-computing approach. In future the execution time can further be reduced by balancing the workload of grid nodes.

Incremental DataGrid Mining Algorithm for Mobility Prediction of Mobile Users

Problem statement: Mobility prediction is the important issue in Personal Communication Systems (PCS). Mobile users moving logs are stored in data grid located in different locations. Distributed data mining algorithm is applied on this moving logs to generate the mobility pattern of mobile users. As new moving logs are added to the data grid, existing mobility pattern becomes invalid and it should be updated. One of the existing work to derive the new mobility pattern is re-executing the algorithm from scratch results in excessive computation. Approach: We had designed new incremental algorithm by maintaining infrequent mobility patterns, which avoids unnecessary scan of full database. Incremental data mining algorithm taken lesser time to compute new mobility patterns. The discovered location patterns can be used to provide various location based services to the mobile user by the application server in mobile computing environment. Data grid provided geographically distributed database for computational grid which implements incremental data mining algorithm. We built data grid system on a cluster of workstation using open source globus toolkit 4.0 and Message Passing Interface extended with Grid Services (MPICH-G2). Results: The experiments were conducted on original data sets and data were added incrementally and the computation time was recorded for each data sets. The performance improvement for increment size of 100 K was about 55% for 0.20% support count and it is increased to 60% for 0.25% support count. The performance is increased about 65% for the support count 0.30%. Conclusion: We analyzed our results with various sizes of data sets and the proof shows the time taken to generate mobility pattern by incremental mining algorithm is less than re-computing approach. In future the execution time can further be reduced by balancing the workload of grid nodes.

Data mining in P2P networks

To analyze both the operational mechanism of current distributed data mining and the characteristics of the P2P technology: non-centralized peer and asynchronism, by extending the iterative process of classical K-mean algorithm, a distributed data mining algorithm was designed in this paper to implement k-mean thinking in a P2P networks. This algorithm exchanges information only between directly connected nodes, and can cluster local data on each peer in a global view. Finally, simulation experiments show that the algorithm is effective and accurate.

Air Pollution Monitoring and Mining Based on Sensor Grid in London

In this paper, we present a distributed infrastructure based on wireless sensors network and Grid computing technology for air pollution monitoring and mining, which aims to develop low-cost and ubiquitous sensor networks to collect real-time, large scale and comprehensive environmental data from road traffic emissions for air pollution monitoring in urban environment. The main informatics challenges in respect to constructing the high-throughput sensor Grid are discussed in this paper. We present a two-layer network framework, a P2P e-Science Grid architecture, and the distributed data mining algorithm as the solutions to address the challenges. We simulated the system in TinyOS to examine the operation of each sensor as well as the networking performance. We also present the distributed data mining result to examine the effectiveness of the algorithm.