MapReduce Programming Framework Research Articles

Development of computer application system and database testing based on data encryption technology

Abstract In this paper, we use the MapReduce programming framework for the development of a computer application system and design the vision module, control module and transaction module of the system based on MVC architecture. Focusing on the encryption design of the database, the data stored in the database is chunked through the Map function, and all the chunking results are aggregated based on the Reduce function, and synchronized encryption of the database is combined with the Paillier homomorphic encryption algorithm. On this basis, the system environment is configured, and the application system developed in this paper is tested, focusing on exploring the stability of the database. The test results show that the decryption output time is 21s when the Paillier encryption algorithm is 200, and the first set of database peak is 76% of the server load, which is within the tolerable range. Security test on the database: the results show that the probability of an attack on all ports with a database security factor less than 6 is around 0.3. There is no significant difference, and the database test is good.

Research on Intelligentization of Cloud Computing Programs Based on Self-awareness

Abstract Through the research of MapReduce programming framework of cloud computing, the current MapReduce program only solves specific problems, and there is no design experience or design feature summary of MapReduce program, let alone formal description and experience inheritance and application of knowledge base. In order to solve the problem of intelligent cloud computing program, a general MapReduce program generation method is designed. This paper proposes the architecture of intelligent cloud computing by studying AORBCO model and combining cloud computing technology. According to the behavior control mechanism in AORBCO model, a program generation method of MapReduce in intelligent cloud computing is proposed. This method will extract entity information in input data set and entity information in knowledge base in intelligent cloud computing for similarity calculation, and extract the entity in the top order as key key-value pair information in intelligent cloud computing judgment data set. The data processing types are divided, and then aligned with each specific MapReduce capability, and the MapReduce program generation experiment is verified in the AORBCO model development platform. The experiment shows that the complexity of big data MapReduce program code is simplified, and the generated code execution efficiency is good.

Prediction and Big Data Impact Analysis of Telecom Churn by Backpropagation Neural Network Algorithm from the Perspective of Business Model.

This study aims to transform the existing telecom operators from traditional Internet operators to digital-driven services, and improve the overall competitiveness of telecom enterprises. Data mining is applied to telecom user classification to process the existing telecom user data through data integration, cleaning, standardization, and transformation. Although the existing algorithms ensure the accuracy of the algorithm on the telecom user analysis platform under big data, they do not solve the limitations of single machine computing and cannot effectively improve the training efficiency of the model. To solve this problem, this article establishes a telecom customer churn prediction model with the help of backpropagation neural network (BPNN) algorithm, and deploys the MapReduce programming framework on Hadoop platform. Using the data of a telecom company, this article analyzes the loss of telecom customers in the big data environment. The research shows that the accuracy of telecom customer churn prediction model in BPNN is 82.12%. After deploying large data sets, the learning and training time of the model is greatly shortened. When the number of nodes is 8, the acceleration ratio of the model remains at 60 seconds. Under big data, the telecom user analysis platform not only ensures the accuracy of the algorithm, but also solves the limitations of single machine computing and effectively improves the training efficiency of the model. Compared with that of the existing research, the accuracy of the model is improved by 25.36%, and the running time is shortened by about twice. This business model based on BPNN algorithm has obvious advantages in processing more data sets, and has great reference value for the digital-driven business model transformation of the telecommunications industry.

IMapC: Inner MAPping Combiner to Enhance the Performance of MapReduce in Hadoop

Hadoop is a framework for storing and processing huge amounts of data. With HDFS, large data sets can be managed on commodity hardware. MapReduce is a programming model for processing vast amounts of data in parallel. Mapping and reducing can be performed by using the MapReduce programming framework. A very large amount of data is transferred from Mapper to Reducer without any filtering or recursion, resulting in overdrawn bandwidth. In this paper, we introduce an algorithm called Inner MAPping Combiner (IMapC) for the map phase. This algorithm in the Mapper combines the values of recurring keys. In order to test the efficiency of the algorithm, different approaches were tested. According to the test, MapReduce programs that are implemented with the Default Combiner (DC) of IMapC will be 70% more efficient than those that are implemented without one. To make computations significantly faster, this work can be combined with MapReduce.

Big Data analytics for Advanced Viticulture

Big data analytics involve systematic approach to find hidden patterns to help the organization grow from large volume and variety of data. In recent years big data analytics is widely used in the agricultural domain to improve yield. Viticulture (the cultivation of grapes) is one of the most lucrative farming in India. It is a subdivision of horticulture and is the study of wine growing. The demand for Indian Wine is increasing at about 27% each year since the 21st century and thus more and more ways are being developed to improve the quality and quantity of the wine products. In this paper, we focus on a specific agricultural practice as viticulture. Weather forecasting and disease detection are the two main research areas in precision viticulture. Leaf disease detection as a part of plant pathology is the key research area in this paper. It can be applied on vineyards of India where farmers are bereft of the latest technologies. Proposed system architecture comprises four modules: Data collection, data preprocessing, classification and visualization. Database module involve grape leaf dataset, consists of healthy images combined with disease leaves such as Black measles, Black rot, and Leaf blight. Models have been implemented on Apache Hadoop using map reduce programming framework. It apply feature extraction to extract various features of the live images and classification algorithm with reduced computational complexity. Gray Level Co-occurrence Matrix (GLCM) followed by K-Nearest Neighborhood (KNN) algorithm. System also recommends the necessary steps and remedies that the viticulturists can take to assure that the grapes can be salvaged at the right time and in the right manner based on classification results. Overall system will help Indian viticulturists to improve the harvesting process. Accuracy of the model is 72% and it can be increased as a future work by including deep learning with time series grape leaf images.

A Data-Driven Analysis of the Paradigm Shift From Permanent to Contractual Recruitment

Big data is the high-volume, high-variety data which involves data storage, data management, and data analysis that presents a wide view of business possibility for real-time data, sensor data, and streaming data over the web. Big data relies on technology, analysis, and mythology where technology deals with computation power, accuracy, linking, and large datasets; analysis is to find patterns by analyzing large datasets to discover hidden information; and mythology is the wrong beliefs that large datasets give insight knowledge of data that is not obtained by small datasets. In this paper, the authors analyzed the major benefits the organization see from employing contract workers using map reduce programming framework.

DRAV: Detection and repair of data availability violations in Internet of Things

The application of the Internet of Things has produced large amounts of data in different scenarios, which are accompanied with problems, such as consistency and integrity violations. Existing research on dealing with data availability violations is insufficient. In this work, the detection and repair of data availability violations (DRAV) framework is proposed to detect and repair data violations in Internet of Things with a distributed parallel computing environment. DRAV uses algorithms in the MapReduce programming framework, and these include detection and repair algorithms based on enhanced conditional function dependency for data consistency violation, MapJoin, and ReduceJoin algorithms based on master data for k-nearest neighbor–based integrity violation detection, and repair algorithms. Experiments are conducted to determine the effect of the algorithms. Results show that DRAV improves data availability in Internet of Things compared with existing methods by detecting and repairing violations.

Efficient processing of all-k-nearest-neighbor queries in the MapReduce programming framework

Numerous modern applications, from social networking to astronomy, need efficient answering of queries on spatial data. One such query is the All k Nearest-Neighbor Query, or k Nearest-Neighbor Join, that takes as input two datasets and, for each object of the first one, returns the k nearest-neighbors from the second one. It is a combination of the k nearest-neighbor and join queries and is computationally demanding. Especially, when the datasets involved fall in the category of Big Data, a single machine cannot efficiently process it. Only in the last few years, papers proposing solutions for distributed computing environments have appeared in the literature. In this paper, we focus on parallel and distributed algorithms using the Apache Hadoop framework. More specifically, we focus on an algorithm that was recently presented in the literature and propose improvements to tackle three major challenges that distributed processing faces: improvement of load balancing (we implement an adaptive partitioning scheme based on Quadtrees), acceleration of local processing (we prune points during calculations by utilizing plane-sweep processing), and reduction of network traffic (we restructure and reduce the output size of the most demanding phase of computation). Moreover, by using real 2D and 3D datasets, we experimentally study the effect of each improvement and their combinations on performance of this literature algorithm. Experiments show that by carefully addressing the three aforementioned issues, one can achieve significantly better performance. Thereby, we conclude to a new scalable algorithm that adapts to the data distribution and significantly outperforms its predecessor. Moreover, we present an experimental comparison of our algorithm against other well-known MapReduce algorithms for the same query and show that these algorithms are also significantly outperformed.

Combustion Optimization for Coal Fired Power Plant Boilers Based on Improved Distributed ELM and Distributed PSO

Increasing the combustion efficiency of power plant boilers and reducing pollutant emissions are important for energy conservation and environmental protection. The power plant boiler combustion process is a complex multi-input/multi-output system, with a high degree of nonlinearity and strong coupling characteristics. It is necessary to optimize the boiler combustion model by means of artificial intelligence methods. However, the traditional intelligent algorithms cannot deal effectively with the massive and high dimensional power station data. In this paper, a distributed combustion optimization method for boilers is proposed. The MapReduce programming framework is used to parallelize the proposed algorithm model and improve its ability to deal with big data. An improved distributed extreme learning machine is used to establish the combustion system model aiming at boiler combustion efficiency and NOx emission. The distributed particle swarm optimization algorithm based on MapReduce is used to optimize the input parameters of boiler combustion model, and weighted coefficient method is used to solve the multi-objective optimization problem (boiler combustion efficiency and NOx emissions). According to the experimental analysis, the results show that the method can optimize the boiler combustion efficiency and NOx emissions by combining different weight coefficients as needed.

Big Data: Tutorial and guidelines on information and process fusion for analytics algorithms with MapReduce

We live in a world were data are generated from a myriad of sources, and it is really cheap to collect and storage such data. However, the real benefit is not related to the data itself, but with the algorithms that are capable of processing such data in a tolerable elapse time, and to extract valuable knowledge from it. Therefore, the use of Big Data Analytics tools provide very significant advantages to both industry and academia. The MapReduce programming framework can be stressed as the main paradigm related with such tools. It is mainly identified by carrying out a distributed execution for the sake of providing a high degree of scalability, together with a fault-tolerant scheme.In every MapReduce algorithm, first local models are learned with a subset of the original data within the so-called Map tasks. Then, the Reduce task is devoted to fuse the partial outputs generated by each Map. The ways of designing such fusion of information/models may have a strong impact in the quality of the final system. In this work, we will enumerate and analyze two alternative methodologies that may be found both in the specialized literature and in standard Machine Learning libraries for Big Data. Our main objective is to provide an introduction of the characteristics of these methodologies, as well as giving some guidelines for the design of novel algorithms in this field of research. Finally, a short experimental study will allow us to contrast the scalability issues for each type of process fusion in MapReduce for Big Data Analytics.

Distributed intrusion detection based on hybrid gene expression programming and cloud computing in a cyber physical power system

With the increasingly widespread application of information and communication technology, the smart grid has gradually evolved into a cyber physical system characterised by deep integration between the information space and physical space. All manner of intrusion attacks on cyber physical power systems are growing more and more frequent. Timely and accurate detection and identification of these intrusions are essential for the effective control and protection of cyber physical power systems. For massive and high-dimensional intrusion behaviour data in cyber physical power systems, distributed intrusion detection based on hybrid gene expression programming and cloud (DID-HGEPCloud) computing is proposed. In the DID-HGEPCloud, attribution reduction with noise data based on rough set and a global intrusion model based on non-linear least squares are applied to improve the efficiency and accuracy of intrusion detection. At the same time, the MapReduce programming framework of cloud computing is adopted, and parallelisation of the model of the proposed algorithm is performed to enhance its ability to manage massive and high-dimensional data. Comparative experiments show that the algorithm proposed in this paper has obvious advantages in terms of false attack rate, DAR, and average time consumed. Furthermore, the proposed algorithm possesses excellent parallel performance.

Why Linguistic Fuzzy Rule Based Classification Systems perform well in Big Data Applications?

The significance of addressing Big Data applications is beyond all doubt. The current ability of extracting interesting knowledge from large volumes of information provides great advantages to both corporations and academia. Therefore, researchers and practitioners must deal with the problem of scalability so that Machine Learning and Data Mining algorithms can address Big Data properly. With this end, the MapRe-duce programming framework is by far the most widely used mechanism to implement fault-tolerant distributed applications. This novel framework implies the design of a divide-and-conquer mechanism in which local models are learned separately in one stage (Map tasks) whereas a second stage (Reduce) is devoted to aggregate all sub-models into a single solution. In this paper, we focus on the analysis of the behavior of Linguistic Fuzzy Rule Based Classification Systems when embedded into a MapRe-duce working procedure. By retrieving different information regarding the rules learned throughout the MapReduce process, we will be able to identify some of the capabilities of this particular paradigm that allowed them to provide a good performance when addressing Big Data problems. In summary, we will show that linguistic fuzzy classifiers are a robust approach in case of scalability requirements.

Applications of the MapReduce programming framework to clinical big data analysis: current landscape and future trends.

The emergence of massive datasets in a clinical setting presents both challenges and opportunities in data storage and analysis. This so called “big data” challenges traditional analytic tools and will increasingly require novel solutions adapted from other fields. Advances in information and communication technology present the most viable solutions to big data analysis in terms of efficiency and scalability. It is vital those big data solutions are multithreaded and that data access approaches be precisely tailored to large volumes of semi-structured/unstructured data.The MapReduce programming framework uses two tasks common in functional programming: Map and Reduce. MapReduce is a new parallel processing framework and Hadoop is its open-source implementation on a single computing node or on clusters. Compared with existing parallel processing paradigms (e.g. grid computing and graphical processing unit (GPU)), MapReduce and Hadoop have two advantages: 1) fault-tolerant storage resulting in reliable data processing by replicating the computing tasks, and cloning the data chunks on different computing nodes across the computing cluster; 2) high-throughput data processing via a batch processing framework and the Hadoop distributed file system (HDFS). Data are stored in the HDFS and made available to the slave nodes for computation.In this paper, we review the existing applications of the MapReduce programming framework and its implementation platform Hadoop in clinical big data and related medical health informatics fields. The usage of MapReduce and Hadoop on a distributed system represents a significant advance in clinical big data processing and utilization, and opens up new opportunities in the emerging era of big data analytics. The objective of this paper is to summarize the state-of-the-art efforts in clinical big data analytics and highlight what might be needed to enhance the outcomes of clinical big data analytics tools. This paper is concluded by summarizing the potential usage of the MapReduce programming framework and Hadoop platform to process huge volumes of clinical data in medical health informatics related fields.

Research on Algorithm of Distributed Reactive Power Optimization Based on Cloud Computing and Improved NSGA-II

A distributed reactive power optimization algorithm is put forward based on cloud computing and improved NSGA-II (fast non-dominated sorting genetic algorithm) in this paper. It is designed to solve problem of multi-objective reactive power optimization with huge amounts of data in power grid, whose difficulties lie in local optimum and slow processing speed. First, NSGA-II's crossover and mutation operator are improved based on Cloud Model, so as to satisfy the adaptive characteristics. In this way, we improved global optimization ability and convergence speed when dealing with large-scale reactive power optimization. Second, we introduced cloud computing, parallelized the proposed algorithm based on MapReduce programming framework. In this way, we achieved distributed improved NSGA-II algorithm, effectively improved the calculation speed of handling massive high-dimensional reactive power optimization. Through theoretical study demonstrated the superiority of the algorithm to solve the Multi-Objective reactive power optimization.

A Multiple Ring Fast Computing Method Based on MapReduce in Digraph

Graph algorithm has been widely used in modern computer network technology, and with the increase of data, graph algorithm is faced with the challenge of big data. Using MapReduce programming framework, this paper puts forward a multiple ring fast computing method in diagraph. Compared with the previous methods, this method is more efficient. And through the experiment the method are compared with the previous method.

Detecting Local Communities within a Large Scale Social Network Using Mapreduce

Social network partitioning has become a very important function. One objective for partitioning is to identify interested communities to target for marketing and advertising activities. The bottleneck to detection of these communities is the large scalability of the social network. Previous methods did not effectively address the problem because they considered the overall network. Social networks have strong locality, so designing a local algorithm to find an interested community to address this objective is necessary. In this paper, we develop a local partition algorithm, named, Personalized PageRank Partitioning, to identify the community. We compute the conductance of the social network with a Personalized PageRank and Markov chain stationary distribution of the social network, and then sweep the conductance to find the smallest cut. The efficiency of the cut can reach. In order to detect a larger scale social network, we design and implement the algorithm on a MapReduce-programming framework. Finally, we execute our experiment on several actual social network data sets and compare our method to others. The experimental results show that our algorithm is feasible and very effective.

GPFS-SNC: An enterprise cluster file system for Big Data

A new class of data-intensive applications commonly referred to as Big Data applications (e.g., customer sentiment analysis based on click-stream logs) involves processing massive amounts of data with a focus on semantically transforming the data. This class of applications is massively parallel and well suited for the MapReduce programming framework that allows users to perform large-scale data analyses such that the application execution layer handles the system architecture, data partitioning, and task scheduling. In this paper, we introduce GPFS-SNC (General Parallel File System for Shared Nothing Clusters), a scalable file system that operates over a cluster of commodity machines and direct-attached storage and meets the requirements of analytics and traditional applications that are typically used together in analytics solutions. The architecture extends an existing enterprise cluster file system to support these emerging classes of workloads by applying five innovative optimizations: 1) locality awareness to allow compute jobs to be scheduled on nodes where the data resides, 2) metablocks that allow large and small block sizes to co-exist in the same file system to meet the needs of different types of applications, 3) write affinity that allows applications to dictate the layout of files on different nodes in order to maximize both write and read bandwidth, 4) pipelined replication to maximize use of network bandwidth for data replication, and 5) distributed recovery to minimize the effect of failures on ongoing computation.