MapReduce Parallel Research Articles

An improved distributed storage and query for remote sensing data

With the rapid development of information technology, the amount of remote sensing data is increasing at an unprecedented scale. In the presence of massive remote sensing data, the traditional processing methods have the problems of low efficiency and lack of scalability, so this paper uses open source big data technology to improve it. Firstly, the storage model of remote sensing image data is designed by using the distributed storage database HBase. Then, the grid index and the Hibert curve are combined to establish the index for the image data. Finally, the method of MapReduce parallel processing is used to write and query remote sensing images. The experimental results show that the method can effectively improve the data writing and query speed, and has good scalability.

An Efficient Parallel Mining Algorithm Representative Pattern Set of Large-Scale Itemsets in IoT

With the advent of the age of big data, people can collect rich and diverse data from a wide variety of collection devices, such as those of the Internet of Things. Knowledge hidden in large data is very useful and valuable. Frequent pattern mining, as a basic method of data mining, is applied to every aspect of society. However, the application of traditional frequent pattern mining methods to big data involves bottlenecks due to the large number of result sets. Such bottlenecks make it difficult to produce practical value in production and life. Therefore, mining representative pattern sets has been proposed. However, most existing algorithms select representative patterns after mining frequent pattern sets. This framework can make the runtime difficult to evaluate in large data environments. To solve the above-mentioned problems, this paper presents an online representative pattern-set parallel-mining algorithm. Within the parallel MapReduce framework, this algorithm uses horizontal segmentation to process the database and then applies the online mining algorithm to mine the locally represented pattern sets on each small database. Finally, several performance optimization strategies are proposed. As shown by numerous experiments on the actual dataset, the algorithm proposed in this paper improves the time efficiency by one order of magnitude. Several optimization strategies reduce the execution time to varying degrees.

Finding Top- $k$ Dominance on Incomplete Big Data Using MapReduce Framework

Incomplete data is one major kind of multi-dimensional dataset that has random-distributed missing nodes in its dimensions. It is very difficult to retrieve information from this type of dataset when it becomes large. Finding top-k dominant values in this type of dataset is a challenging procedure. Some algorithms are present to enhance this process, but most are efficient only when dealing with small incomplete data. One of the algorithms that make the application of top-k dominating (TKD) query possible is the Bitmap Index Guided (BIG) algorithm. This algorithm greatly improves the performance for incomplete data, but it is not designed to find top-k dominant values in incomplete big data. Several other algorithms have been proposed to find the TKD query, such as Skyband Based and Upper Bound Based algorithms, but their performance is also questionable. Algorithms developed previously were among the first attempts to apply TKD query on incomplete data; however, these algorithms suffered from weak performance. This paper proposes MapReduced Enhanced Bitmap Index Guided Algorithm (MRBIG) for dealing with the aforementioned issues. MRBIG uses the MapReduce framework to enhance the performance of applying top-k dominance queries on large incomplete datasets. The proposed approach uses the MapReduce parallel computing approach involving multiple computing nodes. The framework separates the tasks between several computing nodes to independently and simultaneously work to find the result. This method has achieved up to two times faster processing time in finding the TKD query result when compared to previously proposed algorithms.

A Parallel MapReduce Algorithm to Efficiently Support Itemset Mining on High Dimensional Data

Abstract In today's world, large volumes of data are being continuously generated by many scientific applications, such as bioinformatics or networking. Since each monitored event is usually characterized by a variety of features, high-dimensional datasets have been continuously generated. To extract value from these complex collections of data, different exploratory data mining algorithms can be used to discover hidden and non-trivial correlations among data. Frequent closed itemset mining is an effective but computational expensive technique that is usually used to support data exploration. Thanks to the spread of distributed and parallel frameworks, the development of scalable approaches able to deal with the so called Big Data has been extended to frequent itemset mining. Unfortunately, most of the current algorithms are designed to cope with low-dimensional datasets, delivering poor performances in those use cases characterized by high-dimensional data. This work introduces PaMPa-HD, a MapReduce-based frequent closed itemset mining algorithm for high dimensional datasets. An efficient solution has been proposed to parallelize and speed up the mining process. Furthermore, different strategies have been proposed to easily configure the algorithm parameter. The experimental results, performed on real-life high-dimensional use cases, show the efficiency of the proposed approach in terms of execution time, load balancing and robustness to memory issues.

A MapReduce-based improvement algorithm for DBSCAN

This paper proposes an improved adaptive density-based spatial clustering of applications with noise (DBSCAN) algorithm based on genetic algorithm and MapReduce parallel computing programming frame...

MapReduce Based Parallel Bayesian Network for Manufacturing Quality Control

Increasing complexity of industrial products and manufacturing processes have challenged conventional statistics based quality management approaches in the circumstances of dynamic production. A Bayesian network and big data analytics integrated approach for manufacturing process quality analysis and control is proposed. Based on Hadoop distributed architecture and MapReduce parallel computing model, big volume and variety quality related data generated during the manufacturing process could be dealt with. Artificial intelligent algorithms, including Bayesian network learning, classification and reasoning, are embedded into the Reduce process. Relying on the ability of the Bayesian network in dealing with dynamic and uncertain problem and the parallel computing power of MapReduce, Bayesian network of impact factors on quality are built based on prior probability distribution and modified with posterior probability distribution. A case study on hull segment manufacturing precision management for ship and offshore platform building shows that computing speed accelerates almost directly proportionally to the increase of computing nodes. It is also proved that the proposed model is feasible for locating and reasoning of root causes, forecasting of manufacturing outcome, and intelligent decision for precision problem solving. The integration of bigdata analytics and BN method offers a whole new perspective in manufacturing quality control.

Multiagent-consensus-MapReduce-based attribute reduction using co-evolutionary quantum PSO for big data applications

The attribute reduction for big data applications has become an urgent challenge in pattern recognition, machine learning and data mining. In this paper, we introduce the multi-agent consensus MapReduce optimization model and co-evolutionary quantum PSO with self-adaptive memeplexes for designing the attribute reduction method, and propose a multiagent-consensus-MapReduce-based attribute reduction algorithm (MCMAR). Firstly, the co-evolutionary quantum PSO with self-adaptive memeplexes is designed for grouping particles into different memeplexes, which aims to explore the search space and locate the global best region during the attribute reduction of big datasets. Secondly, the four layers neighborhood radius framework with compensatory scheme is constructed to partition big attribute sets by exploiting the interdependency among multiple-relevant-attribute sets. Thirdly, a novel multi-agent consensus MapReduce optimization model is adopted to perform the multiple-relevance-attribute reduction, in which five kinds of agents are used to conduct the ensemble co-evolutionary optimization. So the uniform reduction framework of different agents’ co-evolutionary game under the bounded rationality is further refined. Fourthly, the approximation MapReduce parallelism mechanism is permitted to formalize to the multi-agent co-evolutionary consensus structure, interaction and adaptation, which enhances different agents to share their solutions. Finally, extensive experimental studies substantiate the effectiveness and accuracy of MCMAR on some well-known benchmark datasets. Moreover, successful applications in big medical datasets are expected to dramatically scaling up MCMAR for complex infant brain MRI in terms of efficiency and feasibility.

Extremely high-dimensional optimization with MapReduce: Scaling functions and algorithm

Large scale optimization is an active research area in which many algorithms, benchmark functions, and competitions have been proposed to date. However, extremely high-dimensional optimization problems comprising millions of variables demand new approaches to perform effectively in results quality and efficiently in time. Memetic algorithms are popular in continuous optimization but they are hampered on such extremely large dimensionality due to the limitations of computational and memory resources, and heuristics must tackle the immensity of the search space. This work advances on how the MapReduce parallel programming model allows scaling to problems with millions of variables, and presents an adaptation of the MA-SW-Chains algorithm to the MapReduce framework. Benchmark functions from the IEEE CEC 2010 and 2013 competitions are considered and results with 1, 3 and 10 million variables are presented. MapReduce demonstrates to be an effective approach to scale optimization algorithms on extremely high-dimensional problems, taking advantage of the combined computational and memory resources distributed in a computer cluster.

Pattern matching of signature-based IDS using Myers algorithm under MapReduce framework

The rapid increase in wired Internet speed and the constant growth in the number of attacks make network protection a challenge. Intrusion detection systems (IDSs) play a crucial role in discovering suspicious activities and also in preventing their harmful impact. Existing signature-based IDSs have significant overheads in terms of execution time and memory usage mainly due to the pattern matching operation. Therefore, there is a need to design an efficient system to reduce overhead. This research intends to accelerate the pattern matching operation through parallelizing a matching algorithm on a multi-core CPU. In this paper, we parallelize a bit-vector algorithm, Myers algorithm, on a multi-core CPU under the MapReduce framework. On average, we achieve four times speedup using our multi-core implementations when compared to the serial version. Additionally, we use two implementations of MapReduce to parallelize the Myers algorithm using Phoenix++ and MAPCG. Our MapReduce parallel implementations of the Myers algorithm are compared with an earlier message passing interface (MPI)-based parallel implementation of the algorithm. The results show 1.3 and 1.7 times improvement for Phoenix++ and MAPCG MapReduce implementations over MPI respectively.

Parallel and Scalable Map Reduce and Pipeline Tree Classifiers for Massive Dataset Using Map Reduce and Data Flow Pipeline

Parallel and Scalable Map Reduce and Pipeline Tree Classifiers for Massive Dataset Using Map Reduce and Data Flow Pipeline

A Parallel Adaboost-Backpropagation Neural Network for Massive Image Dataset Classification.

Image classification uses computers to simulate human understanding and cognition of images by automatically categorizing images. This study proposes a faster image classification approach that parallelizes the traditional Adaboost-Backpropagation (BP) neural network using the MapReduce parallel programming model. First, we construct a strong classifier by assembling the outputs of 15 BP neural networks (which are individually regarded as weak classifiers) based on the Adaboost algorithm. Second, we design Map and Reduce tasks for both the parallel Adaboost-BP neural network and the feature extraction algorithm. Finally, we establish an automated classification model by building a Hadoop cluster. We use the Pascal VOC2007 and Caltech256 datasets to train and test the classification model. The results are superior to those obtained using traditional Adaboost-BP neural network or parallel BP neural network approaches. Our approach increased the average classification accuracy rate by approximately 14.5% and 26.0% compared to the traditional Adaboost-BP neural network and parallel BP neural network, respectively. Furthermore, the proposed approach requires less computation time and scales very well as evaluated by speedup, sizeup and scaleup. The proposed approach may provide a foundation for automated large-scale image classification and demonstrates practical value.

MRSPAKE : Hadoop MapReduce를 이용한 웹 규모의 공간 지식 추출기

본 논문에서는 Hadoop MapReduce 병렬 분산 컴퓨팅 환경을 이용해 개발한 공간 지식 추출기를 제안한다. 이 공간 지식 추출기는 대용량의 공간 데이터 집합으로부터, 임의의 두 공간 객체들 사이에 만족되는 위상 관계와 방향 관계를 나타내는 정성 공간 지식 베이스를 생성해낸다. 본 논문에서 제안하는 MapReduce 기반의 대용량 공간 지식 추출기 MRSPAKE는 HDFS 상의 분산 공간 데이터 파일에 대한 R 트리 색인과 범위 질의들을 이용함으로써, 웹 규모의 정성 공간 지식 베이스를 매우 효율적으로 추출해낸다. 대표적인 공개 데이터 집합인 Open Street Map(OSM)을 이용한 성능 분석 실험을 통해, 본 논문에서 제안하는 웹 규모의 공간 지식 추출기 MRSPAKE의 높은 성능과 확장성을 확인할 수 있었다. In this paper, we present a spatial knowledge extractor implemented in Hadoop MapReduce parallel, distributed computing environment. From a large spatial dataset, this knowledge extractor automatically derives a qualitative spatial knowledge base, which consists of both topological and directional relations on pairs of two spatial objects. By using R-tree index and range queries over a distributed spatial data file on HDFS, the MapReduce-enabled spatial knowledge extractor, MRSPAKE, can produce a web-scale spatial knowledge base in highly efficient way. In experiments with the well-known open spatial dataset, Open Street Map (OSM), the proposed web-scale spatial knowledge extractor, MRSPAKE, showed high performance and scalability.

Cooperative ant colony-genetic algorithm based on spark

By taking full advantages of both the map and reduce function for the MapReduce parallel framework and the memory computation for the Spark platform, this paper designs and implements the algorithms for solving the traveling salesman problem based on ant colony algorithm on MapReduce framework and Spark platform. Next, adds the nearest neighbor selection strategy for choosing next city for the Spark platform ant colony algorithm, and combines it with genetic algorithm by using the optimal individual between ant colony algorithm and genetic algorithm, in order to update each other's best individual at the end of each iteration. Experimental results show that with the increase of ant colony size, compared to the stand-alone ant colony algorithm, MapReduce ant colony algorithm reflects the superiority of parallel computation; compared to the MapReduce ant colony algorithm, Spark platform ant colony algorithm reflects the superiority of memory computing. Cooperated with genetic algorithm, the solution has been improved significantly in its precision.

An Efficient Bulk Synchronous Parallelized Scheduler for Bioinformatics Application on Public Cloud

Genomic sequence alignment of varied species is one of the most sort of applications in bioinformatics. In future bioinformatics technologies are expected to produce genomic data of terabyte. Bioinformatics computation require super computer for sequence alignment computation which involves huge cost. Parallelization technique is a way forward in computing sequence alignment with limited cost and time. Cloud computing and MapReduce framework play an important role in bioinformatics intensive application to achieve parallelization since it provides a consistent performance over time and it also provides good fault tolerant mechanism. The existing gene sequencing methodologies are designed based on Hadoop-MapReduce framework which adopts a serial execution strategy which is an area of concern. This work introduces a Smith-Waterman Alignment on the Bulk synchronous Parallel Map Reduce (SW-BSPMR) cloud platform for bioinformatics gene sequence alignment. This work adopts a widely accepted and accurate SW algorithm for sequence alignment and parallel synchronous scheduler methodology of map and reduce framework process is considered. A customized MapReduce based on Microsoft Azure cloud platform is developed to overcome the issue in Hadoop-MapReduce framework. The experimental study presented in this work proves that the SW-BSPMR can accurately and effectively align bioinformatics genomic sequences of various read length.

A Crowdsourcing Worker Quality Evaluation Algorithm on MapReduce for Big Data Applications

Crowdsourcing is a new emerging distributed computing and business model on the backdrop of Internet blossoming. With the development of crowdsourcing systems, the data size of crowdsourcers, contractors and tasks grows rapidly. The worker quality evaluation based on big data analysis technology has become a critical challenge. This paper first proposes a general worker quality evaluation algorithm that is applied to any critical tasks such as tagging, matching, filtering, categorization and many other emerging applications, without wasting resources. Second, we realize the evaluation algorithm in the Hadoop platform using the MapReduce parallel programming model. Finally, to effectively verify the accuracy and the effectiveness of the algorithm in a wide variety of big data scenarios, we conduct a series of experiments. The experimental results demonstrate that the proposed algorithm is accurate and effective. It has high computing performance and horizontal scalability. And it is suitable for large-scale worker quality evaluations in a big data environment.

SSC Based RS: An Efficient Service Recommendation System for Handling Big Data Applications

Objectives: To find an appropriate web service and reduce the time taken for introducing web service, to improve parallel processing, to reduce the complexity and to improve its scalability and efficiency in big data environment. Methods: MapReduce framework in Hadoop platform is for increase the efficiency and scalability in big data domain, SSC based RS, web service information are structured in hierarchical format. The proposed system calculates the semantic comparison between the big data applications. Findings: SSC based RS (Semantic Similarity Calculation Based Recommendation System) is used to efficiently suggest better services for the requested users, by using semantic dictionary the semantic similarity will be calculated. Here, the services are stored in the hierarchical structure will increase the recommendation process faster. An experimental result shows that our proposed algorithm provides a suitable recommended service compared to other existing approaches. Applications/Improvement: In Big Data the proposed technique improves the efficiency and scalability by applying MapReduce parallel processing standard on Hadoop environment.

Big Data: A Parallel Particle Swarm Optimization-Back-Propagation Neural Network Algorithm Based on MapReduce.

A back-propagation (BP) neural network can solve complicated random nonlinear mapping problems; therefore, it can be applied to a wide range of problems. However, as the sample size increases, the time required to train BP neural networks becomes lengthy. Moreover, the classification accuracy decreases as well. To improve the classification accuracy and runtime efficiency of the BP neural network algorithm, we proposed a parallel design and realization method for a particle swarm optimization (PSO)-optimized BP neural network based on MapReduce on the Hadoop platform using both the PSO algorithm and a parallel design. The PSO algorithm was used to optimize the BP neural network’s initial weights and thresholds and improve the accuracy of the classification algorithm. The MapReduce parallel programming model was utilized to achieve parallel processing of the BP algorithm, thereby solving the problems of hardware and communication overhead when the BP neural network addresses big data. Datasets on 5 different scales were constructed using the scene image library from the SUN Database. The classification accuracy of the parallel PSO-BP neural network algorithm is approximately 92%, and the system efficiency is approximately 0.85, which presents obvious advantages when processing big data. The algorithm proposed in this study demonstrated both higher classification accuracy and improved time efficiency, which represents a significant improvement obtained from applying parallel processing to an intelligent algorithm on big data.

Feedback Autonomic Provisioning for Guaranteeing Performance in MapReduce Systems

Companies have a fast growing amounts of data to process and store, a data explosion is happening next to us. Currently one of the most common approaches to treat these vast data quantities are based on the MapReduce parallel programming paradigm. While its use is widespread in the industry, ensuring performance constraints, while at the same time minimizing costs, still provides considerable challenges. We propose a coarse grained control theoretical approach, based on techniques that have already proved their usefulness in the control community. We introduce the first algorithm to create dynamic models for Big Data MapReduce systems, running a concurrent workload. Furthermore, we identify two important control use cases: relaxed performance—minimal resource and strict performance. For the first case we develop two feedback control mechanism. A classical feedback controller and an even-based feedback, that minimises the number of cluster reconfigurations as well. Moreover, to address strict performance requirements a feedforward predictive controller that efficiently suppresses the effects of large workload size variations is developed. All the controllers are validated online in a benchmark running in a real 60 node MapReduce cluster, using a data intensive Business Intelligence workload. Our experiments demonstrate the success of the control strategies employed in assuring service time constraints.

User abnormal behavior analysis based on neural network clustering

It is the premise of accessing and controlling cloud environment to establish the mutual trust relationship between users and clouds. How to identify the credible degree of the user identity and behavior becomes the core problem? This paper proposes a user abnormal behavior analysis method based on neural network clustering to resolve the problems of over-fitting and flooding the feature information, which exists in the process of traditional clustering analysis and calculating similarity. Firstly, singular value decomposition (SVD) is applied to reduce dimension and de-noise for massive data, where map-reduce parallel processing is used to accelerate the computation speed, and neural network model is used for softening points. Secondly, information entropy is added to hidden layer of neural network model to calculate the weight of each attribute. Finally, weight factor is used to calculate the similarity to make the cluster more accuracy. For the problem of analyzing the mobile cloud user behaviors, the experimental results show that the scheme has higher detection speed (DS) and clustering accuracy than traditional schemes. The proposed method is more suitable for the mobile cloud environment.

MRQUTER: MapReduce 프레임워크를 이용한 병렬 정성 시간 추론기

ABSTRACT In order to meet rapid changes of Web information, it is necessary to extend the current Web technologies to represent both the valid time and location of each fact and knowledge, and reason their relationships. Until recently, many researches on qualitative temporal reasoning have been conducted in laboratory-scale, dealing with small knowledge bases. However, in this paper, we propose the design and implementation of a parallel qualitative temporal reasoner, MRQUTER, which can make reasoning over Web-scale large knowledge bases. This parallel temporal reasoner was built on a Hadoop cluster system using the MapReduce parallel programming framework. It decomposes the entire qualitative temporal reasoning process into several MapReduce jobs such as the encoding and decoding job, the inverse and equal reasoning job, the transitive reasoning job, the refining job, and applies some optimization techniques into each component reasoning job implemented with a pair of Map and Reduce functions. Through experiments using large benchmarking temporal knowledge bases, MRQUTER shows high reasoning performance and scalability.Keywords:Qualitative Temporal Reasoning, Disjunctive Relations, Composition Table, MapReduce, Parallel Temporal Reasoner