Elastic MapReduce Research Articles

A powerful heuristic method for generating efficient database systems

Heuristic functions are an integral part of MapReduce software, both in Apache Hadoop and Spark. If the heuristic function performs badly, the load in the reduce part will not be balanced and access times spike. To investigate this problem closer, we run an optimal database program with numerous different heuristic functions on database. We will leverage the Amazon elastic MapReduce framework. The paper investigates on general purpose, implementation, and evaluation of heuristic algorithm for generating optimal database system, checksum, and special heuristic functions. With the analysis, we present the corresponding runtime results. For the coding part, the records counting part is hasty and can only work for local Hadoop part, it can be debugged and optimized for general purpose implement on Hadoop and Spark and turn into an effective performance monitor tool. As mentioned before, there are strange issue, also the performance of BLAKE2s is unexpectedly slow in that it’s widely accepted the performance of BLAKE2s is much better than MD5 and SHA256, we would like to figure out why the common-sense performance of heuristics is deferent from what we got in distributed frameworks.

A Dynamic and Failure-Aware Task Scheduling Framework for Hadoop

Hadoop has become a popular framework for processing data-intensive applications in cloud environments. A core constituent of Hadoop is the scheduler, which is responsible for scheduling and monitoring the jobs and tasks, and rescheduling them in case of failures. Although fault-tolerance mechanisms have been proposed for Hadoop, the performance of Hadoop can be significantly impacted by unforeseen events in the cloud environment. In this paper, we introduce a dynamic and failure-aware framework that can be integrated within Hadoop scheduler and adjust the scheduling decisions based on collected information about the cloud environment. Our framework relies on predictions made by machine learning algorithms and scheduling policies generated by a Markovian Decision Process (MDP), to adjust its scheduling decisions on the fly. Instead of the fixed heartbeat-based failure detection commonly used in Hadoop to track active TaskTrackers (i.e., nodes that process the scheduled tasks), our proposed framework implements an adaptive algorithm that can dynamically detect the failures of the TaskTracker. To deploy our proposed framework, we have built, ATLAS+, an AdapTive Failure-Aware Scheduler for Hadoop. To assess the performance of ATLAS+, we conduct a large empirical study on a 100-node Hadoop cluster deployed on Amazon Elastic MapReduce (EMR), comparing the performance of ATLAS+ with those of three Hadoop schedulers (FIFO, Fair, and Capacity). Results show that ATLAS+ outperforms FIFO, Fair, and Capacity schedulers. ATLAS+ can reduce the number of failed jobs by up to 43 percent and the number of failed tasks by up to 59 percent. On average, ATLAS+ could reduce the total execution time of jobs by 10 minutes, which represents 40 percent of the job execution times, and by up to 3 minutes for tasks, which represents 47 percent of the task execution time. ATLAS+ also reduced CPU and memory usage by 22 and 20 percent, respectively.

Distributed Approach to Process Satellite Image Edge Detection on Hadoop Using Artificial Bee Colony

The remote sensing domain has witnessed tremendous growth in the past decade, due to advancement in technology. In order to store and process such a large amount of data, a platform like Hadoop is leveraged. This article proposes a MapReduce (MR) approach to perform edge detection of satellite images using a nature-inspired algorithm Artificial Bee Colony (ABC). Edge detection is one of the significant steps in the field of image processing and is being used for object detection in the image. The article also compares two edge detection approaches on Hadoop with respect to scalability parameters such as scaleup and speedup. The experiment makes use of Amazon AWS Elastic MapReduce cluster to run MR jobs. It focuses on traditional edge detection algorithms like Canny Edge (CE) and the proposed MR based Artificial Bee Colony approach. It observes that for five images, the scaleup value of CE is 1.1 whereas, for MR-ABC, it is 1.2. Similarly, speedup values come out to be 1.02 and 1.04, respectively. The algorithm proposed by authors in this article scales comparatively better when compared to Canny Edge.

DECA: scalable XHMM exome copy-number variant calling with ADAM and Apache Spark

BackgroundXHMM is a widely used tool for copy-number variant (CNV) discovery from whole exome sequencing data but can require hours to days to run for large cohorts. A more scalable implementation would reduce the need for specialized computational resources and enable increased exploration of the configuration parameter space to obtain the best possible results.ResultsDECA is a horizontally scalable implementation of the XHMM algorithm using the ADAM framework and Apache Spark that incorporates novel algorithmic optimizations to eliminate unneeded computation. DECA parallelizes XHMM on both multi-core shared memory computers and large shared-nothing Spark clusters. We performed CNV discovery from the read-depth matrix in 2535 exomes in 9.3 min on a 16-core workstation (35.3× speedup vs. XHMM), 12.7 min using 10 executor cores on a Spark cluster (18.8× speedup vs. XHMM), and 9.8 min using 32 executor cores on Amazon AWS’ Elastic MapReduce. We performed CNV discovery from the original BAM files in 292 min using 640 executor cores on a Spark cluster.ConclusionsWe describe DECA’s performance, our algorithmic and implementation enhancements to XHMM to obtain that performance, and our lessons learned porting a complex genome analysis application to ADAM and Spark. ADAM and Apache Spark are a performant and productive platform for implementing large-scale genome analyses, but efficiently utilizing large clusters can require algorithmic optimizations and careful attention to Spark’s configuration parameters.

Large scale microbiome profiling in the cloud.

MotivationBacterial metagenomics profiling for metagenomic whole sequencing (mWGS) usually starts by aligning sequencing reads to a collection of reference genomes. Current profiling tools are designed to work against a small representative collection of genomes, and do not scale very well to larger reference genome collections. However, large reference genome collections are capable of providing a more complete and accurate profile of the bacterial population in a metagenomics dataset. In this paper, we discuss a scalable, efficient and affordable approach to this problem, bringing big data solutions within the reach of laboratories with modest resources.ResultsWe developed Flint, a metagenomics profiling pipeline that is built on top of the Apache Spark framework, and is designed for fast real-time profiling of metagenomic samples against a large collection of reference genomes. Flint takes advantage of Spark’s built-in parallelism and streaming engine architecture to quickly map reads against a large (170 GB) reference collection of 43 552 bacterial genomes from Ensembl. Flint runs on Amazon’s Elastic MapReduce service, and is able to profile 1 million Illumina paired-end reads against over 40 K genomes on 64 machines in 67 s—an order of magnitude faster than the state of the art, while using a much larger reference collection. Streaming the sequencing reads allows this approach to sustain mapping rates of 55 million reads per hour, at an hourly cluster cost of $8.00 USD, while avoiding the necessity of storing large quantities of intermediate alignments.Availability and implementation Flint is open source software, available under the MIT License (MIT). Source code is available at https://github.com/camilo-v/flint.Supplementary information Supplementary data are available at Bioinformatics online.

Device Contextual Content Publishing in Media & Publishing Industry using Big Data Analytics on AWS

Media & Publishing industry was traditionally a Paper and Print Industry. Since the revolution of Internet, industry started moving print to the digital form. Ever since the rapid penetration of mobile phones the media industry has rapidly scaled down paper publishing and adopted digital form successfully Internet speeds have also increased the adoption of Digital Print’s. With Newspapers being accessed globally in its digital form, it is extremely important for publishers to keep their content readily accessible and rich for various devices – Tablets, Laptops, Desktop’s, Mobile Phones, Smart Watches, Digital reader’s etc. This Paper talks about an ECONOMICAL & HIGHLY SCALABLE Big Data analytics implementation using AWS Elastic Map Reduce (EMR) to derive trends on end user usage patterns and choice of device. This will help the publishers rapidly scale to provide device contextual content to end users with ever changing access mechanisms

Optimizing the Performance of Clouds Using Hash Codes in Apache Hadoop and Spark

Hash functions are an integral part of MapReduce software, both in Apache Hadoop and Spark. If the hash function performs badly, the load in the reduced part will not be balanced and access times will spike. To investigate this problem further, we ran the Wordcount program with numerous different hash functions on Amazon AWS. In particular, we will leverage the Amazon Elastic MapReduce framework. The paper investigates the general purpose, cryptographic, checksum, and special hash functions. Through the analysis, we present the corresponding runtime results.

Spark for Social Science

Urban has developed an elastic and powerful approach to the analysis of massive datasets using Amazon Web Services’ Elastic MapReduce (EMR) and the Spark framework for distributed memory and processing. The goal of the project is to deliver powerful and elastic Spark clusters to researchers and data analysts with as little setup time and effort possible, and at low cost. To do that, at the Urban Institute, we use two critical components: (1) an Amazon Web Services (AWS) CloudFormation script to launch AWS Elastic MapReduce (EMR) clusters (2) a bootstrap script that runs on the Master node of the new cluster to install statistical programs and development environments (RStudio and Jupyter Notebooks). The Urban Institute’s Spark for Social Science Github page holds code used to setup the cluster and tutorials for learning how to program in R and Python.

A course on big data analytics

This report details a course on big data analytics designed for undergraduate junior and senior computer science students. The course is heavily focused on projects and writing code for big data processing. It is designed to help students learn parallel and distributed computing frameworks and techniques commonly used in industry. The curriculum includes a progression of projects requiring increasingly sophisticated big data processing ranging from data preprocessing with Linux tools, distributed processing with Hadoop MapReduce and Spark, and database queries with Hive and Google’s BigQuery. We discuss hardware infrastructure and experimentally evaluate the cost/benefit of an on-premise server versus Amazon’s Elastic MapReduce. Finally, we showcase outcomes of our course in terms of student engagement and anonymous student feedback.

Distributed simulation optimization and parameter exploration framework for the cloud

Simulation models are becoming an increasingly popular tool for the analysis and optimization of complex real systems in different fields. Finding an optimal system design requires performing a large sweep over the parameter space in an organized way. Hence, the model optimization process is extremely demanding from a computational point of view, as it requires careful, time-consuming, complex orchestration of coordinated executions. In this paper, we present the design of SOF (Simulation Optimization and exploration Framework in the cloud), a framework which exploits the computing power of a cloud computational environment in order to carry out effective and efficient simulation optimization strategies. SOF offers several attractive features. Firstly, SOF requires “zero configuration”, as it does not require any additional software installed on the remote node; only standard Apache Hadoop and SSH access are sufficient. Secondly, SOF is transparent to the user, since the user is totally unaware that the system operates on a distributed environment. Finally, SOF is highly customizable and programmable, since it enables the running of different simulation optimization scenarios using diverse programming languages – provided that the hosting platform supports them – and different simulation toolkits, as developed by the modeler. The tool has been fully developed and is available on a public repository11SOF GitHub public repository, https://github.com/isislab-unisa/sof. under the terms of the open source Apache License. It has been tested and validated on several private platforms, such as a dedicated cluster of workstations, as well as on public platforms, including the Hortonworks Data Platform and Amazon Web Services Elastic MapReduce solution.

HCE<SUB align="right">m model and a comparative workload analysis of Hadoop cluster

This paper describes the HCEm model, designed to estimate the size of a cluster running Hadoop, in a given timeframe on cloud environments. The HCEm consists of a light optimisation layer for MapReduce jobs and a model to estimate the size of a Hadoop cluster. Additionally, this paper presents a comparative study of HCEm using similar applications and workloads in two production Hadoop clusters, the Amazon Elastic MapReduce and a private cloud in a large financial company, in order to evaluate the performance of the model in real and intensive processing environments. The estimates generated by the HCEm model and processing performed are representative and consistent, which can help researchers and engineers understand the workload characteristics of Hadoop clusters in their production environments. The performance differences shown between the real environments, confirmed that the increased sharing of physical computing host resources reduces the accuracy of the model.

Rail-dbGaP: analyzing dbGaP-protected data in the cloud with Amazon Elastic MapReduce.

Motivation: Public archives contain thousands of trillions of bases of valuable sequencing data. More than 40% of the Sequence Read Archive is human data protected by provisions such as dbGaP. To analyse dbGaP-protected data, researchers must typically work with IT administrators and signing officials to ensure all levels of security are implemented at their institution. This is a major obstacle, impeding reproducibility and reducing the utility of archived data.Results: We present a protocol and software tool for analyzing protected data in a commercial cloud. The protocol, Rail-dbGaP, is applicable to any tool running on Amazon Web Services Elastic MapReduce. The tool, Rail-RNA v0.2, is a spliced aligner for RNA-seq data, which we demonstrate by running on 9662 samples from the dbGaP-protected GTEx consortium dataset. The Rail-dbGaP protocol makes explicit for the first time the steps an investigator must take to develop Elastic MapReduce pipelines that analyse dbGaP-protected data in a manner compliant with NIH guidelines. Rail-RNA automates implementation of the protocol, making it easy for typical biomedical investigators to study protected RNA-seq data, regardless of their local IT resources or expertise.Availability and Implementation: Rail-RNA is available from http://rail.bio. Technical details on the Rail-dbGaP protocol as well as an implementation walkthrough are available at https://github.com/nellore/rail-dbgap. Detailed instructions on running Rail-RNA on dbGaP-protected data using Amazon Web Services are available at http://docs.rail.bio/dbgap/.Contacts: anellore@gmail.com or langmea@cs.jhu.eduSupplementary information: Supplementary data are available at Bioinformatics online.

Using adaptive resource allocation to implement an elastic MapReduce framework

SummaryToday, we are observing a transition of science paradigms from the computational science to data‐intensive science. With the exponential increase of input and intermediate data, more applications are developed using the MapReduce programming model, which is regarded as an appropriate programming model for analysing large data sets. A MapReduce framework runs its applications on a cluster, where the computing capacity allocated to the applications is limited and may not fill their runtime resource demand. In this case, the Map/Reduce tasks have to wait in a queues, and the applications suffer from a poor performance. This work develops an autonomic resource manager within the Hadoop MapReduce framework. The manager is capable of getting aware of the overloading or under‐loading situations with the resources allocated to its user community. For the former, it takes an action of requesting more resources from, for example, the batch system of a High Performance Computing (HPC) cluster or Computing Clouds and integrates the additional resources, in case of acquisition, into the Hadoop MapReduce runtime. For the latter, the manager gives the free resources back to its source. We extended the existing Hadoop MapReduce resource manager to implement the proposed strategy and validated the concept on an HPC cluster with standard benchmark applications. Experimental results show a significant performance gain, for example, an up to 45% improvement in execution time for running multiple applications. Copyright © 2016 John Wiley & Sons, Ltd.

Leveraging MapReduce to efficiently extract associations between biomedical concepts from large text data

Large biomedical text data represents an important source of information that not only enables researchers to discover in-depth knowledge about biological systems, but also helps healthcare professionals do evidence-based medicine in clinical settings. However, investigating and analyzing these data is often both data-intensive and computation-intensive. In this paper, we investigate how to use MapReduce, a parallel and distributed programming paradigm, to efficiently mine the associations between biomedical concepts extracted from a large set of biomedical articles. First, biomedical concepts were obtained by matching text to Unified Medical Language System (UMLS) Metathesaurus, a biomedical vocabulary and standard database. Then we developed a MapReduce algorithm that could be used to calculate a category of interestingness measures defined on the basis of a 2×2 contingency table. This algorithm consists of two MapReduce jobs and takes a stripes approach to reduce the number of intermediate results. Experiments were conducted using Amazon Elastic MapReduce (EMR) with an input of 33,960 articles from TREC (Text REtrieval Conference) 2006 Genomics Track. Performance test indicated that our algorithm had approximately linear scalability and was more efficient than a ‘‘pairs’’ approach in the literature. The physician in our project team evaluated a subset of the association mining results related to drug-disease treatment and found that meaningful association rules ranked high.

HCE<SUB align="right">m model and a comparative workload analysis of Hadoop cluster

This paper describes the HCEm model, designed to estimate the size of a cluster running Hadoop, in a given timeframe on cloud environments. The HCEm consists of a light optimisation layer for MapReduce jobs and a model to estimate the size of a Hadoop cluster. Additionally, this paper presents a comparative study of HCEm using similar applications and workloads in two production Hadoop clusters, the Amazon Elastic MapReduce and a private cloud in a large financial company, in order to evaluate the performance of the model in real and intensive processing environments. The estimates generated by the HCEm model and processing performed are representative and consistent, which can help researchers and engineers understand the workload characteristics of Hadoop clusters in their production environments. The performance differences shown between the real environments, confirmed that the increased sharing of physical computing host resources reduces the accuracy of the model.

Energy Cost Aware Scheduling of MapReduce Jobs across Geographically Distributed Nodes

MapReduce framework is designed to distribute computations among a large set of nodes. MapReduce implementation is typically designed to operate on nodes within a single cluster or data center, like Amazon’s Elastic MapReduce. However, there are benefits if one can choose a set of cloud providers, and use geographically distributed private and public clouds to execute a MapReduce job in a geo-distributed environment. In this poster, we present a technique to choose geographically distributed nodes for executing a MapReduce job with the objective of minimizing the total energy cost of completing the job while satisfying Quality of Service (QoS). We consider the MapReduce system in a geographically distributed environment that consists of Nd data nodes, Nm mapper nodes, and Nr reducer nodes. Each data node Di (1 ≤ i ≤ Nd) has an amount of input data di. Each data node is connected to every mapper node Mj (1 ≤ j ≤ Nd). Each mapper node j is connected to each reducer node Rk (1 ≤ k ≤ Nr). The compute rate of each mapper node Mj and each reducer node Rk are different. The electricity prices vary according to the location of MapReduce nodes [1]. The push, map, shuffle, and reduce phases are executed sequentially to complete a MapReduce job. We assume that there is a global barrier between the phases, which requires all nodes in one phase to complete execution before the execution at any node in the next phase can proceed. In our design, when a MapReduce job is submitted, we schedule all the necessary MapReduce nodes assuming that the regional electricity prices, the compute rates of MapReduce nodes, the bandwidth of links are known. The MapReduce user specifies the deadline constraint T that satisfies

Hadoop Based Data Intensive Computation on IaaS Cloud Platforms

Cloud computing is a relatively new form of computing, which uses virtualized resources and is dynamically scalable and is often provided as pay for use service over the Internet or Intranet or both. With increasing demand for data storage in the cloud, study of data intensive applications is becoming a primary focus. Data intensive applications are those which involve a high CPU usage, processsing large volumes of data typically in size of hundreds of gigabytes, terabytes, or petabytes. This study was conducted on Amazon's Elastic Cloud Compute (EC2) and Amazon Elastic Map Reduce (EMR) using HiBench Hadoop Benchmark Suite. HiBench is a Hadoop benchmark suite and is used for performing and evaluating Hadoop based data intensive computation on both these cloud paltforms. Both quantitative and qualitative comparison was performed on both Amazon EC2 and Amazon EMR, including a study of their pricing models and measures are suggested for future studies and research.

Ontology Based Document Clustering Using MapReduce

Nowadays, document clustering is considered as a data intensive task due to the dramatic, fast increase in the number of available documents. Nevertheless, the features that represent those documents are also too large. The most common method for representing documents is the vector space model, which represents document features as a bag of words and does not represent semantic relations between words. In this paper we introduce a distributed implementation for the bisecting k-means using MapReduce programming model. The aim behind our proposed implementation is to solve the problem of clustering intensive data documents. In addition, we propose integrating the WordNet ontology with bisecting k-means in order to utilize the semantic relations between words to enhance document clustering results. Our presented experimental results show that using lexical categories for nouns only enhances internal evaluation measures of document clustering; and decreases the documents features from thousands to tens features. Our experiments were conducted using Amazon Elastic MapReduce to deploy the Bisecting k-means algorithm.

Toward protecting control flow confidentiality in cloud-based computation

Cloud based computation services have grown in popularity in recent years. Cloud users can deploy an arbitrary computation cluster to public clouds and execute their programs on that remote cluster to reduce infrastructure investment and maintenance costs. However, how to leverage cloud resources while keeping the computation confidential is a new challenge to be explored. In this paper, we propose runtime control flow obfuscation (RCFO) to protect the control flow confidentiality of outsourced programs. RCFO transforms an outsourced program into two parts: the public program running on the untrusted public cloud and the private program running on the trusted private cloud. By hiding parts of the control flow information in the private program and inserting fake branch statements into the public program, RCFO raises the bar for static and dynamic analysis-based reverse engineering attacks. Based on RCFO, we implement a system called MRDisguiser to protect cloud-based MapReduce services. We perform experiments on a real MapReduce service, Amazon Elastic MapReduce. The experimental results indicate that MRDisguiser is compatible with current cloud-based MapReduce services, and incurs moderate performance overhead. Specifically, when the obfuscation degree increases from 0 to 1.0, the average performance overhead is between 14.9% and 33.2%.

Processing Shotgun Proteomics Data on the Amazon Cloud with the Trans-Proteomic Pipeline

Cloud computing, where scalable, on-demand compute cycles and storage are available as a service, has the potential to accelerate mass spectrometry-based proteomics research by providing simple, expandable, and affordable large-scale computing to all laboratories regardless of location or information technology expertise. We present new cloud computing functionality for the Trans-Proteomic Pipeline, a free and open-source suite of tools for the processing and analysis of tandem mass spectrometry datasets. Enabled with Amazon Web Services cloud computing, the Trans-Proteomic Pipeline now accesses large scale computing resources, limited only by the available Amazon Web Services infrastructure, for all users. The Trans-Proteomic Pipeline runs in an environment fully hosted on Amazon Web Services, where all software and data reside on cloud resources to tackle large search studies. In addition, it can also be run on a local computer with computationally intensive tasks launched onto the Amazon Elastic Compute Cloud service to greatly decrease analysis times. We describe the new Trans-Proteomic Pipeline cloud service components, compare the relative performance and costs of various Elastic Compute Cloud service instance types, and present on-line tutorials that enable users to learn how to deploy cloud computing technology rapidly with the Trans-Proteomic Pipeline. We provide tools for estimating the necessary computing resources and costs given the scale of a job and demonstrate the use of cloud enabled Trans-Proteomic Pipeline by performing over 1100 tandem mass spectrometry files through four proteomic search engines in 9 h and at a very low cost.