MapReduce Cluster Research Articles

YARN Schedulers for Hadoop MapReduce Jobs: Design Goals, Issues and Taxonomy

Objective: Big Data processing is a demanding task, and several big data processing frameworks have emerged during recent decades. The performance of these frameworks greatly dependent on resource management models. Methods: YARN is one of such models which acts as a resource management layer and provides computational resources for execution engines (Spark, MapReduce, storm, etc.) through its schedulers. The most important aspect of resource management is job scheduling. Results: In this paper, we first present the design goal of YARN real-life schedulers (FIFO, Capacity, and Fair) for the MapReduce engine. Later, we discuss the scheduling issues of the Hadoop MapReduce cluster. Conclusion: Many efforts have been carried out in the literature to address issues of data locality, heterogeneity, straggling, skew mitigation, stragglers and fairness in Hadoop MapReduce scheduling. Lastly, we present the taxonomy of different scheduling algorithms available in the literature based on some factors like environment, scope, approach, objective and addressed issues.

A Parallel Fractional Lion Algorithm for Data Clustering Based on MapReduce Cluster Framework

This work introduces a parallel clustering algorithm by modifying the existing Fractional Lion Algorithm (FLA). The proposed work replaces the conventional Euclidean distance measure with the Bhattacharya distance measure to newly propose the improved FLA (IMR-FLA). The proposed IMR-FLA is implemented in both the mapper and the reducer in the MapReduce framework to achieve the parallel clustering. The experimentation of the proposed IMR-FLA is done by using six standard databases, namely Pima Indian diabetes dataset, Heart disease dataset, Hepatitis dataset, localization dataset, breast cancer dataset, and skin segmentation dataset, from the UCI repository. The proposed IMR-FLA has the overall improved Jaccard coefficient value of 0.9357, 0.6572, 0.7462, 0.5944, 0.9418, and 0.8680, for each dataset. Similarly, the proposed IMR-FLA algorithm has outclassed other classifiers' performance with the clustering accuracy value of 0.9674, 0.9471, 0.9677, 0.777, 0.9023, and 0.9585, respectively, for the experimental databases.

Enhancing Leakage Prevention for MapReduce

When public clouds become the platform of choice for MapReduce processing, users are placing higher demands on the privacy of the job data and program. A number of solutions employed trusted hardware to protect MapReduce tasks. However, existing works pointed out that simply protecting individual nodes in the MapReduce cluster with trusted hardware and protecting cross-node communication with encryption still leak information from side-channels. Specifically, attackers can derive data information by observing and manipulating cross-node communication traffic volumes. Although existing works proposed some solutions to prevent such leakage, in this paper, we show that previous solutions still leak critical job information. Additionally, our study shows that previous solutions have limitations from other aspects, including data restriction, partition function restriction, reliability issue, and high overheads. To address all the discovered limitations, we introduced the Strong Shuffle solution. Our analysis and experimental results showed that our solution has reduced the information leakage and addressed other discovered limitations. To support Strong Shuffle, we proposed a variant Bloom Filter, named Group-based Dynamic Bloom Filter (GDBF). Our theoretical analysis showed that GDBF has lower performance and storage overhead than the traditional Scalable Bloom Filter.

RETRACTED ARTICLE: Urban ecological environment investigation based on a cloud computing platform and optimization of computer neural network algorithm

The rapid development of Internet technology has contributed to the development of the Internet. The rapid expansion of the industrial application scale and the sharp increase of data generated by an industrial application and cloud computing have promoted the birth and development of cloud computing platform technology. Data analysis has become an important support for company decision-making; how to quickly and effectively extract useful information from large data sets is very important. Among them, clustering analysis and computer neural network algorithms are the most important methods of data mining. Due to the limitations of computer performance and programming models, traditional data mining methods cannot meet the efficiency and computing needs of a large amount of information. The development of cloud computing technology has opened up a new research field of cloud mining clustering analysis and neural network algorithm formation. The first research is the deployment of the Hadoop cluster based on Linux, and the novelty lies in the parallelization of the MapReduce cluster algorithm. Because there are many clustering algorithms, this paper starts with the K-means clustering algorithm and optimizes the earlier algorithm according to the MapReduce programming model. Through the text clustering process of wine data set in the UCI database, the parallel algorithm is applied to the Hadoop cloud computing platform. Experiments show that the parallel K-means clustering algorithm in MapReduce can significantly improve the execution speed. This paper applies it to the urban ecological environment survey in China, clarifies the relationship between various factors of urban environmental governance, expands the content of urban environmental governance in China, and improves the ecological environment of Chinese cities.

Effective Scheduler for Distributed DNN Training Based on MapReduce and GPU Cluster

Parallel training accelerates the Deep Neural Networks (DNN) training by parallel GPUs. While the in-memory data transmission becomes the cross-node network transmission due to distribution of GPUs on different nodes, which drags the training time. Most researches address it by reducing the data size on network links. However, the factor of network distance is ignored. In this paper, we construct a distributed DNN training architecture based on MapReduce. The customized scheduler is designed to make the computations nodes that finish the training closer to the nodes that storage data. At the same time, the parallel training models are synchronized by adjusting the data transmission time. The experimental results show that the shortened network distance benefits the reduced network traffic usage. The resulting data transmission time decreases the training time by at least 50% and guarantees the synchronization for the parallel training.

Towards Greening MapReduce Clusters Considering Both Computation Energy and Cooling Energy

Increased processing power of MapReduce clusters generally enhances performance and availability at the cost of substantial energy consumption that often incurs higher operational costs (e.g., electricity bills) and negative environmental impacts (e.g., carbon dioxide emissions). There exist a few greening methods for computing clusters in the literature that focus mainly on computational energy consumption leaving cooling energy, which occupies a significant portion of the total energy consumed by the clusters. To this extent, in this article, we propose a machine learning-based approach that reduces the total energy consumption of a MapReduce cluster considering both computational energy and cooling energy. Our approach predicts the number of machines that results in minimum total energy consumption. We perform the prediction through applying different machine learning techniques over year-long data collected from a real setup. We evaluate performance of our approach through both real test-bed experimentation and simulation. Our evaluation reveals that our approach achieves substantial reduction in total energy consumption compared to other state-of-the-art alternatives while experiencing marginal performance degradation in a few cases.

Evaluating the Effects of Modern Storage Devices on the Efficiency of Parallel Machine Learning Algorithms

Big Data analytics is presently one of the most emerging areas of research for both organizations and enterprises. The requirement for deployment of efficient machine learning algorithms over huge amounts of data led to the development of parallelization frameworks and of specialized libraries (like Mahout and MLlib) which implement the most important among these algorithms. Moreover, the recent advances in storage technology resulted in the introduction of high-performing devices, broadly known as Solid State Drives (SSDs). Compared to the traditional Hard Drives (HDDs), SSDs offer considerably higher performance and lower power consumption. Motivated by these appealing features and the growing necessity for efficient large-scale data processing, we compared the performance of several machine learning algorithms on MapReduce clusters whose nodes are equipped with HDDs, SSDs, and devices which implement the latest 3D XPoint technology. In particular, we evaluate several dataset preprocessing methods like vectorization and dimensionality reduction, two supervised classifiers, Naive Bayes and Linear Regression, and the popular k-Means clustering algorithm. We use an experimental cluster equipped with the three aforementioned storage devices under different configurations, and two large datasets, Wikipedia and HIGGS. The experiments showed that the benefits which derive from the usage of SSDs depend on the cluster setup and the nature of the applied algorithms.

Distributed Approach to Process Satellite Image Edge Detection on Hadoop Using Artificial Bee Colony

The remote sensing domain has witnessed tremendous growth in the past decade, due to advancement in technology. In order to store and process such a large amount of data, a platform like Hadoop is leveraged. This article proposes a MapReduce (MR) approach to perform edge detection of satellite images using a nature-inspired algorithm Artificial Bee Colony (ABC). Edge detection is one of the significant steps in the field of image processing and is being used for object detection in the image. The article also compares two edge detection approaches on Hadoop with respect to scalability parameters such as scaleup and speedup. The experiment makes use of Amazon AWS Elastic MapReduce cluster to run MR jobs. It focuses on traditional edge detection algorithms like Canny Edge (CE) and the proposed MR based Artificial Bee Colony approach. It observes that for five images, the scaleup value of CE is 1.1 whereas, for MR-ABC, it is 1.2. Similarly, speedup values come out to be 1.02 and 1.04, respectively. The algorithm proposed by authors in this article scales comparatively better when compared to Canny Edge.

Designing a MapReduce performance model in distributed heterogeneous platforms based on benchmarking approach

MapReduce framework is an effective method for big data parallel processing. Enhancing the performance of MapReduce clusters, along with reducing their job execution time, is a fundamental challenge to this approach. In fact, one is faced with two challenges here: how to maximize the execution overlap between jobs and how to create an optimum job scheduling. Accordingly, one of the most critical challenges to achieving these goals is developing a precise model to estimate the job execution time due to the large number and high volume of the submitted jobs, limited consumable resources, and the need for proper Hadoop configuration. This paper presents a model based on MapReduce phases for predicting the execution time of jobs in a heterogeneous cluster. Moreover, a novel heuristic method is designed, which significantly reduces the makespan of the jobs. In this method, first by providing the job profiling tool, we obtain the execution details of the MapReduce phases through log analysis. Then, using machine learning methods and statistical analysis, we propose a relevant model to predict runtime. Finally, another tool called job submission and monitoring tool is used for calculating makespan. Different experiments were conducted on the benchmarks under identical conditions for all jobs. The results show that the average makespan speedup for the proposed method was higher than an unoptimized case.

Computationally Efficient Simulation of Queues: The R Package queuecomputer

Large networks of queueing systems model important real-world systems such as MapReduce clusters, web-servers, hospitals, call centers and airport passenger terminals. To model such systems accurately, we must infer queueing parameters from data. Unfortunately, for many queueing networks there is no clear way to proceed with parameter inference from data. Approximate Bayesian computation could offer a straightforward way to infer parameters for such networks if we could simulate data quickly enough. We present a computationally efficient method for simulating from a very general set of queueing networks with the R package queuecomputer. Remarkable speedups of more than 2 orders of magnitude are observed relative to the popular DES packages simmer and simpy. We replicate output from these packages to validate the package. The package is modular and integrates well with the popular R package dplyr. Complex queueing networks with tandem, parallel and fork/join topologies can easily be built with these two packages together. We show how to use this package with two examples: a call center and an airport terminal.

Energy aware task scheduling using hybrid firefly - GA in big data

Task scheduling is important of research in big data and it is made in two traditions user level and system level; in user level issues with scheduling between the service provider and customer; in system level issues in scheduling with resources management in the data centre. The drawbacks of various existing methods to increase in power consumption of data centres have become a significant issue. Now the MapReduce clusters constitute a major piece of the data centre for big data applications. Simply the absolute size, high fault-tolerant nature and low utilisation levels make them less energy efficient. The complexity of scheduling increases when there is an increase in the size of the task, it becomes very tedious to perform scheduling effectively. The drawback with existing scheduling algorithm generates higher computational cost and less efficient; the multi-objective scheduling with cloud computing makes it difficult to resolve the problem in the case of complex tasks. These are the primary drawbacks of several existing works, which prompt us to manage this research on task scheduling in cloud computing.

Energy aware task scheduling using hybrid firefly - GA in big data

Task scheduling is important of research in big data and it is made in two traditions user level and system level; in user level issues with scheduling between the service provider and customer; in system level issues in scheduling with resources management in the data centre. The drawbacks of various existing methods to increase in power consumption of data centres have become a significant issue. Now the MapReduce clusters constitute a major piece of the data centre for big data applications. Simply the absolute size, high fault-tolerant nature and low utilisation levels make them less energy efficient. The complexity of scheduling increases when there is an increase in the size of the task, it becomes very tedious to perform scheduling effectively. The drawback with existing scheduling algorithm generates higher computational cost and less efficient; the multi-objective scheduling with cloud computing makes it difficult to resolve the problem in the case of complex tasks. These are the primary drawbacks of several existing works, which prompt us to manage this research on task scheduling in cloud computing.

A Comprehensive View of Scheduling Algorithms for MapReduce Framework in Hadoop

We reside in information world which makes great difficulty in quantifying the aggregate volume of information put away electronically, and exist in the unit of Zettabytes or Exabyte, hence termed as Big Data. They can be unstructured, organized or semi organized, processing of these data cannot able to be done with normal data management techniques. Hadoop framework is utilized to process huge datasets in a proficient and economical way. MapReduce system is utilized to gather information according to the solicitation. To handle expansive volume of information legitimate planning is obliged to accomplish more noteworthy execution. Scheduling jobs in parallel across the nodes is an important concern in MapReduce clusters. Also to improve the performance with the large volume of data proper scheduling method should be adopted. The objective of the work is to study about the different scheduling techniques used in Hadoop environment and to achieve effective fairness without starvation.

Deadline-Aware MapReduce Job Scheduling with Dynamic Resource Availability

As MapReduce is becoming ubiquitous in large-scale data analysis, many recent studies have shown that the performance of MapReduce could be improved by different job scheduling approaches, e.g., Fair Scheduler and Capacity Scheduler. However, most exiting MapReduce job schedulers focus on the scenario that MapReduce cluster is stable and pay little attention to the MapReduce cluster with dynamic resource availability. In fact, MapReduce cluster resources may fluctuate as there is a growing number of Hadoop clusters deployed on hybrid systems, e.g., infrastructure powered by mix of traditional and renewable energy, and cloud platforms hosting heterogeneous workloads. Thus, there is a growing need for providing predictable services to users who have strict requirements on job completion times in such dynamic environments. In this paper, we propose, RDS , a Resource and Deadline-aware Hadoop job Scheduler that takes future resource availability into consideration when minimizing job deadline misses. We formulate the job scheduling problem as an online optimization problem and solve it using an efficient receding horizon control algorithm. To aid the control, we design a self-learning model to estimate job completion times. We further extend the design of RDS scheduler to support flexible performance goals in various dynamic clusters. In particular, we use flexible deadline time bounds instead of the single fixed job completion deadline. We have implemented RDS in the open-source Hadoop implementation and performed evaluations with various benchmark workloads. Experimental results show that RDS substantially reduces the penalty of deadline misses by at least 36 and 10 percent compared with Fair Scheduler and Earliest Deadline First (EDF) scheduler, respectively. In a Hadoop cluster running partially on renewable energy, the experimental result shows the green power based resource prediction approach can further reduce the penalty of deadline misses by 16 percent compared to Auto-Regressive Integrated Moving Average (ARIMA) prediction approach.

Spark for Social Science

Urban has developed an elastic and powerful approach to the analysis of massive datasets using Amazon Web Services’ Elastic MapReduce (EMR) and the Spark framework for distributed memory and processing. The goal of the project is to deliver powerful and elastic Spark clusters to researchers and data analysts with as little setup time and effort possible, and at low cost. To do that, at the Urban Institute, we use two critical components: (1) an Amazon Web Services (AWS) CloudFormation script to launch AWS Elastic MapReduce (EMR) clusters (2) a bootstrap script that runs on the Master node of the new cluster to install statistical programs and development environments (RStudio and Jupyter Notebooks). The Urban Institute’s Spark for Social Science Github page holds code used to setup the cluster and tutorials for learning how to program in R and Python.

A Predictive Map Task Scheduler for Optimizing Data Locality in MapReduce Clusters

This article describes how data locality is becoming one of the most critical factors to affect performance of MapReduce clusters because of network bisection bandwidth becomes a bottleneck. Task scheduler assigns the most appropriate map tasks to nodes. If map tasks are scheduled to nodes without input data, these tasks will issue remote I/O operations to copy the data to local nodes that decrease execution time of map tasks. In that case, prefetching mechanism can be useful to preload the needed input data before tasks is launching. Therefore, the key challenge is how this article can accurately predict the execution time of map tasks to be able to use data prefetching effectively without any data access delay. In this article, it is proposed that a Predictive Map Task Scheduler assigns the most suitable map tasks to nodes ahead of time. Following this, a linear regression model is used for prediction and data locality based algorithm for tasks scheduling. The experimental results show that the method can greatly improve both data locality and execution time of map tasks.

A MapReduce implementation of posterior probability clustering and relevance models for recommendation

Relevance-Based Language Models are a formal probabilistic approach for explicitly introducing the concept of relevance in the Statistical Language Modelling framework. Recently, they have been determined as a very effective way of computing recommendations. When combining this new recommendation approach with Posterior Probabilistic Clustering for computing neighbourhoods, the item ranking is further improved, radically surpassing rating prediction recommendation techniques. Nevertheless, in the current landscape where the number of recommendation scenarios reaching the big data scale is increasing day after day, high figures of effectiveness are not enough. In this paper, we address one urging and common need of recommendation systems which is algorithm scalability. Particularly, we adapted those highly effective algorithms to the functional MapReduce paradigm, that has been previously proved as an adequate tool for enabling recommenders scalability. We evaluated the performance of our approach under realistic circumstances, showing a good scalability behaviour on the number of nodes in the MapReduce cluster. Additionally, as a result of being able to execute our algorithms distributively, we can show measures in a much bigger collection supporting the results presented on the seminal paper.

Prediction-Based and Locality-Aware Task Scheduling for Parallelizing Video Transcoding Over Heterogeneous MapReduce Cluster

MapReduce is a popular programming model in cloud computing to deal with the high computational task, such as video transcoding. It splits the video (task) into multiple segments (subtasks) and transcodes them in parallel in cluster. Due to the complexity of video transcoding and the poor performance of heterogeneous MapReduce cluster, scheduling these subtasks to minimize the total transcoding time is still a challenge. In this paper, we propose a prediction-based and locality-aware task scheduling (PLTS) method for parallelizing video transcoding over heterogeneous MapReduce cluster. First, we analyze video decoding and encoding technologies and predict the segment transcoding complexity, which can provide a foundational base for the following scheduling. Second, we attempt to schedule subtasks on machines that contain the related input data, which are referred to as data locality, so as to reduce large-scale data movement and data transfer during the mapping phase. Third, we formulate the scheduling as a job shop scheduling problem and propose a heuristic PLTS algorithm. It combines the benefits of two traditional heuristic scheduling algorithms, Max–Min and Min–Min, to make load balancing in cluster and short the total transcoding time. The experimental results also show the efficiency of our algorithm.

Two-Stage Performance Engineering of Container-based Virtualization

Cloud computing has become a compelling paradigm built on compute and storage virtualization technologies. The current virtualization solution in the Cloud widely relies on hypervisor-based technologies. Given the recent booming of the container ecosystem, the container-based virtualization starts receiving more attention for being a promising alternative. Although the container technologies are generally considered to be lightweight, no virtualization solution is ideally resource-free, and the corresponding performance overheads will lead to negative impacts on the quality of Cloud services. To facilitate understanding container technologies from the performance engineering's perspective, we conducted two-stage performance investigations into Docker containers as a concrete example. At the first stage, we used a physical machine with “just-enough” resource as a baseline to investigate the performance overhead of a standalone Docker container against a standalone virtual machine (VM). With findings contrary to the related work, our evaluation results show that the virtualization's performance overhead could vary not only on a feature-by-feature basis but also on a job-to-job basis. Moreover, the hypervisor-based technology does not come with higher performance overhead in every case. For example, Docker containers particularly exhibit lower QoS in terms of storage transaction speed. At the ongoing second stage, we employed a physical machine with “fair-enough” resource to implement a container-based MapReduce application and try to optimize its performance. In fact, this machine failed in affording VM-based MapReduce clusters in the same scale. The performance tuning results show that the effects of different optimization strategies could largely be related to the data characteristics. For example, LZO compression can bring the most significant performance improvement when dealing with text data in our case. (Less)

The optimization for recurring queries in big data analysis system with MapReduce

As data-intensive cluster computing systems like MapReduce grow in popularity, there is a strong need to promote the efficiency. Recurring queries, repeatedly being executed for long periods of time on rapidly evolving data-intensive workloads, have become a bedrock component in big data analytic applications. Consequently, this paper presents optimization strategies for recurring queries for big data analysis. Firstly, it analyzes the impact of recurring queries efficiency by MapReduce recurring queries model. Secondly, it proposes the MapReduce consistent window slice algorithm, which can not only create more opportunities for reuse of recurring queries, but also greatly reduce redundant data while loading input data by the fine-grained scheduling. Thirdly, in terms of data scheduling, it designs the MapReduce late scheduling strategy that improve data processing and optimize computation resource scheduling in MapReduce cluster. Finally, it constructs the efficient data reuse execution plans by MapReduce recurring queries reuse strategy. The experimental results on a variety of workloads show that the algorithms outperform the state-of-the-art approaches.