MapReduce Parallel Research Articles

Big Data analysis of demand side management for Industrial IOT applications

Companies today demand even more high quality services and infrastructure in the field ofIndustrial Internet IOT. There are several problems facing smart urbanization. Of special concern are secured energy demand side management. The IIoT makes ransomware, cyber attacks and other security threats open to industrial systems. IIoT will provide meaningful solutions to these problems through the amalgamation of big data analytics. That first author recommends a safe, reliable Demand Side Managementengine for a smart cooperative client using IIoT based Big Data Analysis. A highly distributed framework to the theoretical engine for optimizing DSM across a Home Area Network. A payload authentication framework based on a lightweight handshake mechanism is used to improve the protection of this engine. In order to make users to control different server native resources energy efficiently, our theoretical method uses the portable utilizes of the Constrained Application Protocol. Moreover, data streams with parallel MapReduce processing are managed using big data analysis. In addition, we use Apache Spark with Apache Hadoop to check the theoretical engine to test our input transform match on stable datafile. The test results indicate that the architecture outlined provides useful insights into the intelligent social communities of the IIoT.

Application of Deep Learning in Power load Analysis

Aiming at the problems of slow model training speed and poor prediction effect of traditional power load prediction algorithm, a parallel load prediction method based on deep learning is proposed. The method is based on the MapReduce parallel calculating framework, and the deep belief network model, which is used to parallel training the sample data with the historical load and the weather information, and the model of the training model to predict the load value. The experimental results show that the average root-mean-square error between the predicted power load value and the actual value of the prediction method in this paper is 2.86%. The prediction accuracy is higher than the traditional method, and the training and prediction time are effectively reduced, which can adapt to the prediction requirements of large-scale power data.

Partition Selection for Large-Scale Data Management Using KNN Join Processing

For the data processing with increasing avalanche under large datasets, the k nearest neighbors (KNN) algorithm is a particularly expensive operation for both classification and regression predictive problems. To predict the values of new data points, it can calculate the feature similarity between each object in the test dataset and each object in the training dataset. However, due to expensive computational cost, the single computer is out of work to deal with large-scale dataset. In this paper, we propose an adaptive vKNN algorithm, which adopts on the Voronoi diagram under the MapReduce parallel framework and makes full use of the advantages of parallel computing in processing large-scale data. In the process of partition selection, we design a new predictive strategy for sample point to find the optimal relevant partition. Then, we can effectively collect irrelevant data, reduce KNN join computation, and improve the operation efficiency. Finally, we use a large number of 54-dimensional datasets to conduct a large number of experiments on the cluster. The experimental results show that our proposed method is effective and scalable with ensuring accuracy.

Application of MapReduce parallel association mining on IDS in cloud computing environment

The rise of the cloud computing model has resulted in more than terabytes of data being stored in the cloud platform every day on the Internet. Mining valuable information from these massive data has become an emerging industry direction, but the current Intrusion-detection system (IDS) has been unable to adapt to large-scale log information mining. Therefore, an association rule mining algorithm based on MapReduce parallel computing framework is proposed. Firstly, the frequent itemsets mining algorithm Apriori is analyzed, and the MapReduce model is used to parallelize and improve it to more efficiently complete the mining of frequent itemsets. Secondly, the parallel Apriori is designed to run on IDS. Finally, the simulation experiment was carried out by building an open source cloud computing framework Hadoop cluster. Finally, the simulation experiment was carried out by building an open source cloud computing framework Hadoop cluster. The results show that the proposed method has higher detection efficiency when processing massive data, and requires less processing time.

A distributed WND-LSTM model on MapReduce for short-term traffic flow prediction

Building data-driven intelligent transportation is a significant task for establishing data-centric smart cities, and exceptionally efficient and accurate traffic flow prediction (TFP) is a crucial technology in constructing intelligent transportation systems (ITSs). To address the computation and storage problems of processing traffic flow big data with the centralized model on a traditional mining platform, we propose a distributed long short-term memory weighted model combined with a time window and normal distribution based on a MapReduce parallel processing framework in this paper, named as WND-LSTM. More specifically, under the Hadoop distributed computing platform, a distributed modeling framework of forecasting traffic flow on MapReduce is developed to solve the existing issues of storage and calculation in handling large-scale traffic flow data with the stand-alone learning model. Moreover, a distributed WND-LSTM model is presented on the MapReduce-based distributed modeling framework to enhance the accuracy, efficiency, and scalability of short-term TFP. Finally, we forecast the traffic flow on the Sanlihe East Road of Beijing in China using the proposed WND-LSTM model with the real-world taxi trajectory big data. In particular, the extensively experimental results from a case study demonstrate that the MAPE value of WND-LSTM is 88.48%, 65.79%, 70.46%, 68.21%, 66.95%, 68.43%, and 70.41% lower than that of the autoregressive integrated moving average (ARIMA), logistical regression (LR), support vector regression (SVR), k-nearest neighbor (KNN), stacked autoencoders (SAEs), gated recurrent unit (GRU), and long short-term memory (LSTM), respectively, and achieves 71.25% accuracy improvement on average.

Real-Time Optimization of Dynamic Speed Scaling for Distributed Data Centers

This paper proposes a new distributed real-time optimization for MapReduce-style framework in emerging cloud platforms supporting dynamic speed scaling functions. Distinctively different from the existing MapReduce parallelism strategy with fixed specific data chuck sizes, the new approach is able to dynamically dispatch input data of adequate sizes and synthesize interim processing results according to applications and the state of data centers (DCs). The key idea is to decouple the optimizations of data dispatching, processing, and result aggregation without loss of optimality, by employing stochastic optimization techniques. Another important aspect is that we optimize the subproblem of data processing to leverage the energy- and speed-configurability of DCs, by optimally deciding the number of servers to be activated at every DC and the CPU speeds of the activated servers. Evident from extensive simulations, the proposed approach is able to increase the throughput-cost ratio by up to 94.3 $\%$ , as compared to existing initiatives, and substantially improve the throughput in the case of high-rate data streams.

GenSeg and MR-GenSeg: A Novel Segmentation Algorithm and its Parallel MapReduce Based Approach for Identifying Genomic Regions With Copy Number Variations

Identifying intragenic as well as intergenic sequences of the DNA, having structural alterations, is a significantly important research area, since this may be the root cause of many neurological and autoimmune diseases, including cancer. Working with whole genome NGS data has provided a new insight in this regard, but has lead to huge explosion of data that is growing exponentially. Hence, the challenges lie in efficient means of storage and processing this big data. In this study, we have developed a novel segmentation algorithm, called GenSeg, and its parallel MapReduce based algorithm, called MR-GenSeg, for detecting copy number variations. In order to annotate CNVs (variants), segments formed by GenSeg/MR-GenSeg have been represented in a novel way using a binary tree, where each node is a CNV event. GenSeg considers each position specific data of whole genome DNA sequence, so that precise identification of breakpoints is possible. GenSeg/MR-GenSeg has been compared with twelve popular CNV detection algorithms, where it has outperformed the others in terms of sensitivity, and has achieved a good F-score value. MR-GenSeg has excelled in terms of SpeedUp, when compared with these algorithms. The effect of CNVs on immunoglobulin (IG) genes has also been analysed in this study. Availability: The source codes are available at https://github.com/rituparna-sinha/MapReduce-GENSEG.

Research on Herb Pairs of Classical Formulae of ZHANG Zhong-Jing Using Big Data Technology

ObjectiveTo analyze various herbal combinations in Treatise on Exogenous Febrile Diseases (Shang Han Lun,《伤寒论》) and Synopsis of Prescriptions of the Golden Chamber (Jin Gui Yao Lve, 《金匮要略》), seeking to identify fundamental rules dictating the selection of herbal combinations through probability models and big data technology. MethodsA total of 252 formulae were collected from Treatise on Exogenous Febrile Diseases (Shang Han Lun, 《伤寒论》) and Synopsis of Prescriptions of the Golden Chamber (Jin Gui Yao Lve, 《金匮要略》) by ZHANG Zhong-Jing. Formulae were then preprocessed with all herb names standardized. The concepts of candidate herb pair and candidate herb pair probability were proposed to analyze the rules of combinations in classical formulae based on probability statistics. MapReduce parallel computing framework of distributed big data technology was adopted to analyze large data samples combined with inverted index algorithm. ResultsThe results showed that the core herbs were Glycyrrhizae Radix Rhizoma (Gan Cao, 甘 草 ), Cinnamomi Ramulus (Gui Zhi, 桂 枝 ), Zingiberis Rhizoma Recens (Sheng Jiang, 生姜), Jujubae Fructus (Da Zao, 大枣), Paeoniae Radix Alba (Bai Shao, 白 芍 ), etc. 43 high-frequency pairs co-occurring 10 times or above were extracted, and 35 of these combinations were recognized as traditional herb pairs, such as Cinnamomi Ramulus (Gui Zhi, 桂枝) - Glycyrrhizae Radix Rhizoma (Gan Cao, 甘草), Zingiberis Rhizoma Recens (Sheng Jiang, 生姜) - Jujubae Fructus (Da Zao, 大枣), and Cinnamomi Ramulus (Gui Zhi, 桂枝) - Ginseng Radix Et Rhizoma (Ren Shen, 人参). The other 8 pairs of combinations, such as Paeoniae Radix Alba (Bai Shao, 白芍) - Zingiberis Rhizoma Recens (Sheng Jiang, 生姜), Paeoniae Radix Alba (Bai Shao, 白 芍 ) - Jujubae Fructus (Da Zao, 大 枣 ), and Zingiberis Rhizoma Recens (Sheng Jiang, 生姜) - Ginseng Radix Et Rhizoma (Ren Shen, 人参), were not defined traditionally, but in connection with commonly used herbs. Classical formulae took the core herbs as principles, focusing on tonifying deficiency, strengthening the spleen and the stomach, strengthening the healthy Qi, and eliminating pathogenic factors. The compatibility pattern of properties involved was mainly acrid and sweet, which reflected the compatibility laws of benefiting Qi and tonifying Yang, replenishing Qi and nourishing blood, etc. ConclusionsThe research of classical formulae provides common understanding of some basic rules that have been adopted to tackle common illnesses/diseases using herbal medicine. The results help to reinforce theoretical understanding and development of traditional Chinese medicine (TCM), and revealing the hidden rules of combination in TCM data. Analyzing wider data samples of various herbal combinations through computation and big data technology can further optimize the use of TCM.

Big Data Analytics for Images in Public Cloud using Map Reduce on Local Clusters

MapReduce is a programming model used for parallel computing of big data in public cloud. Big Data have characteristics like variety, velocity and volume. The research work carries out MapReduce using Matlab which is a powerful image processing and numeric computation tool. The research considers unstructured image data in public cloud Dropbox as big data and applies MapReduce algorithm to map and reduce all the images stored in it. The research work aims to retrieve the images in public cloud with maximum Red, Green, Blue color and the colors that intersect between them. The same code is modified to find all Red, Green and Blue that supports more parallelism and aids in improving the speed of MapReduce by eliminating the dependency between iterations. The speed of parallel MapReduce shows considerable improvement only with increased file size and coding style. Parallel MapReduce computation is carried out with default workers, three and four workers of the local cluster with scale up architecture. This model is developed using Matlab and can be implemented in Hadoop as well.

Highly Parallel Map Reduce Process and Efficient Job Scheduling Methodologies of Big Data Systems.

This paper studies about various job scheduling methodologies used in big data systems. Map reduce is a highly efficient distributed job processing strategy for big data systems. Job scheduling is a critical task of any big data system as the volume of jobs need to be processed is tremendous. This study will go over the map reduce process in detail. It also reviews various job scheduling methodologies and tries to perform an efficient comparison among these methodologies.

A Novel Strategy for Retrieving Large Scale Scene Images Based on Emotional Feature Clustering

Due to complicated data structure, image can present rich information, and so images are applied widely at different fields. Although the image can offer a lot of convenience, handling such data consume much time and multi-dimensional space. Especially when users need to retrieve some images from larger-scale image datasets, the disadvantage is more obvious. So, in order to retrieve larger-scale image data effectively, a scene images retrieval strategy based on the MapReduce parallel programming model is proposed. The proposed strategy first, investigates how to effectively store large-scale scene images under a Hadoop cluster parallel processing architecture. Second, a distributed feature clustering algorithm MeanShift is introduced to implement the clustering process of emotional feature of scene images. Finally, several experiments are conducted to verify the effectiveness and efficiency of the proposed strategy in terms of different aspects such as retrieval accuracy, speedup ratio and efficiency and data scalability.

Makespan Map Reduce Architecture for Efficient Memory Utilization

Makespan is referred to the total execution time taken to process the tasks or the jobs to the completion time.The High performance infrastructure in cloud computing provides extensive applications. These applications are preferred in Big Data. The existing Hadoop Map Reduce network incurs the input output memory overhead. The parallel Map Reduce network provides a parallel scheme to reduce makespan times in computing environments. The outcome provides improvement in the coefficient correlation and makespan time. The various challenges in computing dataset is handling large dataset efficiently and providing large amount of datasets with ease. The comprehensive method is to enhance data analyzation techniques. Massive large amount of data which are spread across the large number of machines needs to be parallelized.

A Hadoop Processing Method for Massive Sensor Network Data Based on Internet of Things

Based on the analysis of the architecture of the Internet of Things service platform and the key technologies of cloud computing, a massive sensing information processing scheme based on the Internet of Things service platform is proposed. The scheme first proposes a system architecture model that can satisfy the massive sensor information processing in an open platform environment, and designs multiple functional unit modules of the system. By combining these functional units, service configurability can be realized, facing thousands of services and Tenant. Then, Hadoop open source framework is used to realize the distributed computing of the system, which makes full use of the processing advantages of MapReduce computing model, HBase distributed database and HDFS distributed file system in Hadoop framework, and uses Oracle database as a supplement to realize the system high. Finally, the mass sensor information was deployed and tested. The effectiveness of the Hadoop processing method was verified by analyzing the results of MapReduce parallel computing experiments. The average cache hit rate is 93.1%, which has a high cache hit rate, greatly reduces MySQL database I/O, and improves system performance.

Positive and negative association rule mining in Hadoop\u2019s MapReduce environment

In this paper, we present a Hadoop implementation of the Apriori algorithm. Using Hadoop’s distributed and parallel MapReduce environment, we present an architecture to mine positive as well as negative association rules in big data using frequent itemset mining and the Apriori algorithm. We also analyze and present the results of a few optimization parameters in Hadoop’s MapReduce environment as it relates to this algorithm. The results are presented based on the number of rules generated as well as the run-time efficiency. We find that, a higher amount of parallelization, which means larger block sizes, will increase the run-time efficiency of the Hadoop implementation of the Apriori algorithm.

Advantages of Giraph over Hadoop in Graph Processing

This article presents a comparison of the computing performance of the MapReduce tool Hadoop and Giraph on large-scale graphs. The main ideas of MapReduce and bulk synchronous parallel (BSP) are reviewed as big data computing approaches to highlight their applicability in large-scale graph processing. This paper reviews the execution performance of Hadoop and Giraph on the PageRank algorithm to classify web pages according to their relevance, and on a few other algorithms to find the minimum spanning tree in a graph with the primary goal of finding the most efficient computing approach to work on large-scale graphs. Experimental results show that the use of Giraph for processing large-size graphs reduces the execution time by 25% in comparison with the results obtained using the Hadoop for the same experiments. Giraph represents the optimal option thanks to its in-memory computing approach that avoids secondary memory direct interaction.

DrugR+: A comprehensive relational database for drug repurposing, combination therapy, and replacement therapy

Drug repurposing or repositioning, which introduces new applications of the existing drugs, is an emerging field in drug discovery scope. To enhance the success rate of the research and development (R&D) process in a cost- and time-effective manner, a number of pharmaceutical companies worldwide have made tremendous investments. Besides, many researchers have proposed various methods and databases for the repurposing of various drugs. However, there is not a proper and well-organized database available. To this end, for the first time, we developed a new database based on DrugBank and KEGG data, which is named “DrugR+”. Our developed database provides some advantages relative to the DrugBank, and its interface supplies new capabilities for both single and synthetic repositioning of drugs. Moreover, it includes four new datasets which can be used for predicting drug-target interactions using supervised machine learning methods. As a case study, we introduced novel applications of some drugs and discussed the obtained results. A comparison of several machine learning methods on the generated datasets has also been reported in the Supplementary File. Having included several normalized tables, DrugR + has been organized to provide key information on data structures for the repurposing and combining applications of drugs. It provides the SQL query capability for professional users and an appropriate method with different options for unprofessional users. Additionally, DrugR + consists of repurposing service that accepts a drug and proposes a list of potential drugs for some usages. Taken all, DrugR+ is a free web-based database and accessible using (http://www.drugr.ir), which can be updated through a map-reduce parallel processing method to provide the most relevant information.

Improved support vector machine classification algorithm based on adaptive feature weight updating in the Hadoop cluster environment.

An image classification algorithm based on adaptive feature weight updating is proposed to address the low classification accuracy of the current single-feature classification algorithms and simple multifeature fusion algorithms. The MapReduce parallel programming model on the Hadoop platform is used to perform an adaptive fusion of hue, local binary pattern (LBP) and scale-invariant feature transform (SIFT) features extracted from images to derive optimal combinations of weights. The support vector machine (SVM) classifier is then used to perform parallel training to obtain the optimal SVM classification model, which is then tested. The Pascal VOC 2012, Caltech 256 and SUN databases are adopted to build a massive image library. The speedup, classification accuracy and training time are tested in the experiment, and the results show that a linear growth tendency is present in the speedup of the system in a cluster environment. In consideration of the hardware costs, time, performance and accuracy, the algorithm is superior to mainstream classification algorithms, such as the power mean SVM and convolutional neural network (CNN). As the number and types of images both increase, the classification accuracy rate exceeds 95%. When the number of images reaches 80,000, the training time of the proposed algorithm is only 1/5 that of traditional single-node architecture algorithms. This result reflects the effectiveness of the algorithm, which provides a basis for the effective analysis and processing of image big data.

RETRACTED: Distributed vision navigation technology for mobile robots based on jitter privacy ant colony algorithm

To enhance the real-time obstacle detection technique of mobile robots, a real-time obstacle detection technique of mobile robots based on MapReduce parallel ant colony algorithm was proposed in th...

Detección de anomalías en grandes volúmenes de datos

El desarrollo de la era digital ha traído como consecuencia un incremento considerable de los volúmenes de datos. A estos grandes volúmenes de datos se les ha denominado big data ya que exceden la capacidad de procesamiento de sistemas de bases de datos convencionales. Diversos sectores consideran varias oportunidades y aplicaciones en la detección de anomalías en problemas de big data. Para realizar este tipo de análisis puede resultar muy útil el empleo de técnicas de minería de datos porque permiten extraer patrones y relaciones desde grandes cantidades de datos. El procesamiento y análisis de estos volúmenes de datos, necesitan de herramientas capaces de procesarlos como Apache Spark y Hadoop. Estas herramientas no cuentan con algoritmos específicos para la detección de anomalías. El objetivo del trabajo es presentar un nuevo algoritmo para la detección de anomalías basado en vecindad para de problemas big data. A partir de un estudio comparativo se seleccionó el algoritmo KNNW por sus resultados, con el fin de diseñar una variante big data. La implementación del algoritmo big data se realizó en la herramienta Apache Spark, utilizando el paradigma de programación paralela MapReduce. Posteriormente se realizaron diferentes experimentos para analizar el comportamiento del algoritmo con distintas configuraciones. Dentro de los experimentos se compararon los tiempos de ejecución y calidad de los resultados entre la variante secuencial y la variante big data. La variante big data obtuvo mejores resultados con diferencia significativa. Logrando que la variante big data, KNNW-BigData, pueda procesar grandes volúmenes de datos.

Parallel computing of KNN Query in road network based on MapReduce

K-nearest neighbor (KNN) query algorithm based on road network plays an important role in location based service, which had been widely used in intelligent transportation, roadside assistance and other fields. However, as road network density increases and the number of points of interest increases, query efficiency decreases sharply.In order to improve the query efficiency, we adopted the MapReduce parallel computing framework to complete the query of K neighbor moving objects by designing Map, Reduce, Combiner and other functions. Before the start of the query, the road network was divided into pieces, and each fragment was calculated. The final K-nearest neighbor moving objects were obtained by aggregating the calculated results of each slice to realize the parallel optimization of KNN algorithm based on road network. The experimental results showed that the performance of parallel KNN algorithm based on MapReduce was better than that of serial KNN query algorithm in large-scale road network environment and the larger K value of query request.