MapReduce Programming Model Research Articles

Горизонтальне та вертикальне масштабування методів машинного навчання

The main stages of Machine Learning Pipelines are considered in the paper, such as: train data collection and storage, training and scoring. The effect of the Big Data phenomenon on each of the stages is discussed. Different approaches to efficient organization of computation are on each of the stage are evaluated. In the first part of the paper we introduce the notion of horizontal and vertical scalability together with corresponding cons and pros. We consider some limitations of scaling, such as Amdahl's law. In the second part of the paper we consider scalability of data storage routines. First we discuss relational databases and scalability limitations related to ACID guarantees, which such database satisfy. Then we consider horizontally scalable non-relational databases, so called NoSQL databases. We formulate CAP-theorem as a fundamental limitation of horizontally scalable databases. The third part of the paper is dedicated to scalability of computation based on the MapReduce programming model. We discuss some implementations of this programming model, such as Hadoop and Spark together with some basic principles which they are based on. In the fourth part of the article we consider various approaches towards scaling of Machine Learning methods. We give the general statement of Machine Learning problem. Then we show how MapReduce programming model can be applied for horizontal scaling of Machine Learning methods on the example of Bayessian pattern recognition procedure. On the example of Deep Neural Networks we discuss Machine Learning methods which are not horizontally scalable. Then we consider some approaches towards vertical scaling of such methods based on GPU’s and the TensorFlow programming model.

The Improved SIFT Algorithm Based on Rectangular Operator and Its Parallel Implementation

The SIFT algorithm is one of the most widely used algorithm which bases on local feature extraction. But it could not meet the requirement of the real-time application due to the high time complexity and low execution efficiency. In order to improve these drawback, the authors optimized the SIFT algorithm by using the Gaussian convolution scale of adaptive scale space. The authors also provided the executive process of the improved SIFT algorithm on the MapReduce programming model and compared its performance in terms of the stand-alone and cluster environment. The experiment result showed that compared to the traditional algorithm, the improved algorithm had high execution efficiency, good speedup, scalability and is suitable for massive amounts of image data processing.

DISTRIBUTED INDEXING: PERFORMANCE ANALYSIS OF SOLR, TERRIER AND KATTA INFORMATION RETRIEVALS

Information Retrieval (IR) systems are currently facing a continuous challenge due to the increasing size of datasets. Extremely, large data from different aspects is gathered each day resulting in huge increase in the scale of the raw data available across the internet. Indexing, as the main function of IR systems, is becoming a time-consuming problem. Therefore, efficient indexing of a large volume of data is now a critical requirement of modern IR systems. High performance indexing is performed nowadays over the use of MapReduce programming model. MapReduce is a programming paradigm that enables massive processing and large collections distribution across multiple (hundreds or thousands) commodity computers. To shed some light on this issue, this paper presents a detailed performance analysis of distributed indexing for Solr, Katta and Terrier with the context of MapReduce. In particular, this study compares and analyzes the distributed indexing performance of three frameworks using 1GB, 3GB, 6GB, and 9GB subsets of TREC dataset as the processing power increase. The experiments measure the indexing average time, then throughput, speedup, and efficiency of indexing process. The results show that, Terrier performance is the best in the presence of large collections and scalable processing power. While, Solr performance is the best when having limited computing power and small document collections. Finally, the experimental results show that, Katta produced the worst indexing average time among the three frameworks but its speedup scales linearly with processing power and collection size

Cloud-BS: A MapReduce-based bisulfite sequencing aligner on cloud.

In recent years, there have been many studies utilizing DNA methylome data to answer fundamental biological questions. Bisulfite sequencing (BS-seq) has enabled measurement of a genome-wide absolute level of DNA methylation at single-nucleotide resolution. However, due to the ambiguity introduced by bisulfite-treatment, the aligning process especially in large-scale epigenetic research is still considered a huge burden. We present Cloud-BS, an efficient BS-seq aligner designed for parallel execution on a distributed environment. Utilizing Apache Hadoop framework, Cloud-BS splits sequencing reads into multiple blocks and transfers them to distributed nodes. By designing each aligning procedure into separate map and reducing tasks while an internal key-value structure is optimized based on the MapReduce programming model, the algorithm significantly improves alignment performance without sacrificing mapping accuracy. In addition, Cloud-BS minimizes the innate burden of configuring a distributed environment by providing a pre-configured cloud image. Cloud-BS shows significantly improved bisulfite alignment performance compared to other existing BS-seq aligners. We believe our algorithm facilitates large-scale methylome data analysis. The algorithm is freely available at https://paryoja.github.io/Cloud-BS/ .

COMBImage: a modular parallel processing framework for pairwise drug combination analysis that quantifies temporal changes in label-free video microscopy movies

BackgroundLarge-scale pairwise drug combination analysis has lately gained momentum in drug discovery and development projects, mainly due to the employment of advanced experimental-computational pipelines. This is fortunate as drug combinations are often required for successful treatment of complex diseases. Furthermore, most new drugs cannot totally replace the current standard-of-care medication, but rather have to enter clinical use as add-on treatment. However, there is a clear deficiency of computational tools for label-free and temporal image-based drug combination analysis that go beyond the conventional but relatively uninformative end point measurements.ResultsCOMBImage is a fast, modular and instrument independent computational framework for in vitro pairwise drug combination analysis that quantifies temporal changes in label-free video microscopy movies. Jointly with automated analyses of temporal changes in cell morphology and confluence, it performs and displays conventional cell viability and synergy end point analyses. The image processing algorithms are parallelized using Google’s MapReduce programming model and optimized with respect to method-specific tuning parameters. COMBImage is shown to process time-lapse microscopy movies from 384-well plates within minutes on a single quad core personal computer.This framework was employed in the context of an ongoing drug discovery and development project focused on glioblastoma multiforme; the most deadly form of brain cancer. Interesting add-on effects of two investigational cytotoxic compounds when combined with vorinostat were revealed on recently established clonal cultures of glioma-initiating cells from patient tumor samples. Therapeutic synergies, when normal astrocytes were used as a toxicity cell model, reinforced the pharmacological interest regarding their potential clinical use.ConclusionsCOMBImage enables, for the first time, fast and optimized pairwise drug combination analyses of temporal changes in label-free video microscopy movies. Providing this jointly with conventional cell viability based end point analyses, it could help accelerating and guiding any drug discovery and development project, without use of cell labeling and the need to employ a particular live cell imaging instrument.

An optimal big data processing for smart grid based on hybrid MDM/R architecture to strengthening RE integration and EE in datacenter

Supply chain is a hard business area, where you need to have a perfect balance between demand and supply, day in and day out, by an intricate system that sits underneath it all. Achieving such a system by meeting the ambitious targets of the agreement on climate change can only be achieved through an effective combination of energy efficiency and renewable energy integration. The uncertainty and variability of renewable energy generation can pose challenges for grid operators and can requires additional actions to balance the system. Significant researches which aim is to improve energy efficiency of data center indicates that operating reserves could be procured from many complex and costly techniques. In this paper, we investigate the problem from scheduling of workloads in a data center in order to minimize its energy consumption budget, minimize the conventional grid dependence, and maximize the renewable energy provided to data center, by the ability to temporarily delay or degrade service, with a modified supply-following algorithm. This algorithm attempts to align power consumption with the amount of wind power available, while minimizing the time by which jobs exceed their deadlines. Modification of the algorithm has been performed in the direction of big data processing (wind trace, workload requests, prices, …), servers management. This modification is performed by jobs classification into predefined classes using the classification and regression trees algorithm. New hybrid architecture that manages the Meter Data Management Repository MDM/R was introduced using MapReduce programming model for ETL process and Massive Parallel Processing Database for requests which strongly influences the accuracy and the speediness of the scheduler.

Evaluation of high-level query languages based on MapReduce in Big Data

MapReduce (MR) is a criterion of Big Data processing model with parallel and distributed large datasets. This model knows difficult problems related to low-level and batch nature of MR that gives rise to an abstraction layer on the top of MR. Therefore; several High-Level MapReduce Query Languages built on the top of MR provide more abstract query languages and extend the MR programming model. These High-Level MapReduce Query Languages remove the burden of MR programming away from the developers and make a soft migration of existing competences with SQL skills to Big Data. This paper investigates the very used—common High-Level MapReduce Query Languages built directly on the top of MR that translate queries into executable native MR jobs. It evaluates the performance of the four presented High-Level MapReduce Query Languages: JAQL, Hive, Big SQL and Pig, with regards to their insightful perspectives and ease of programming. The baseline metrics reported are increasing input size, scale-out number of nodes and controlling number of reducers. The experimental results study the technical advantages and limitations of each High-Level MapReduce Query Languages. Finally, the paper provides a summary for developers to choose the High-Level MapReduce Query Languages which fulfill their needs and interests.

Time series decomposition and predictive analytics using MapReduce framework

The recent development in precision agriculture, a large amount of data are generated by site-specific weather stations which will demand a platform for the processing and predictive weather analytics. The sophisticated methodology to solve large amount of data handling problem and process data in a small time is important. In this study, future conditions are predicted from weather stations large data by proposing the predictive approaches based on time series and neural network using MapReduce programming model. We have proposed predictive analytics approaches including the modules, i.e., analysis and decomposition, classification, and prediction. The time series based decomposition approach is proposed to decompose and find out the trend, regular and sophisticated components. The linear components are handled by time series MapReduce based Autoregressive Integrated Moving Average (M-ARIMA) model and nonlinear components are handled by M-K-Nearest Neighbors (M-KNN) model. In addition, the MapReduce-based Hybrid Model (M-HM) was proposed which will use the advantages of time series and neural network to increase prediction accuracy. The study verifies the effectiveness of proposed model over the regular and randomness component of the data. The performance measures and statistical test are performed to validate and check data consistency. In addition, excellent speed-up, scale-up, and size-up were tested by changing the size of data set. However, when the data size increases, the average execution time is reduced by using the MapReduce-based approach over the multiple-node workers.

TRAFFIC MANAGEMENT USING MAPREDUCE FOR BIG DATA ANALYTICS

The MapReduce programming model simplifies large-scale processing on trade goods cluster by exploiting parallel map tasks and cut back tasks. though several efforts are created to boost the performance of MapReduce jobs, they ignore the network traffic generated within the shuffle part, that plays a crucial role in performance improvement.Historically, a hash perform is employed to partition intermediate knowledge among cut back tasks, which, however, isn't traffic-efficient as a result of configuration and knowledge size related to every key don't seem to be taken into thought. During this paper, we have a tendency to study to scale back network traffic value for a MapReduce job by coming up with a unique intermediate knowledge partition theme. what is more, we have a tendency to conjointly contemplate the aggregation placement downside, wherever every aggregation will cut back united traffic from multiple map tasks. A decomposition-based distributed algorithmic rule is projected to subsume the large-scale improvement downside for large knowledge application and a web algorithmic rule is additionally designed to regulate knowledge partition and aggregation in a very dynamic manner. Finally, in depth simulation results demonstrate that our proposals will considerably cut back network traffic value below each offline and on-line cases.

HSRA: Hadoop-based spliced read aligner for RNA sequencing data.

Nowadays, the analysis of transcriptome sequencing (RNA-seq) data has become the standard method for quantifying the levels of gene expression. In RNA-seq experiments, the mapping of short reads to a reference genome or transcriptome is considered a crucial step that remains as one of the most time-consuming. With the steady development of Next Generation Sequencing (NGS) technologies, unprecedented amounts of genomic data introduce significant challenges in terms of storage, processing and downstream analysis. As cost and throughput continue to improve, there is a growing need for new software solutions that minimize the impact of increasing data volume on RNA read alignment. In this work we introduce HSRA, a Big Data tool that takes advantage of the MapReduce programming model to extend the multithreading capabilities of a state-of-the-art spliced read aligner for RNA-seq data (HISAT2) to distributed memory systems such as multi-core clusters or cloud platforms. HSRA has been built upon the Hadoop MapReduce framework and supports both single- and paired-end reads from FASTQ/FASTA datasets, providing output alignments in SAM format. The design of HSRA has been carefully optimized to avoid the main limitations and major causes of inefficiency found in previous Big Data mapping tools, which cannot fully exploit the raw performance of the underlying aligner. On a 16-node multi-core cluster, HSRA is on average 2.3 times faster than previous Hadoop-based tools. Source code in Java as well as a user’s guide are publicly available for download at http://hsra.dec.udc.es.

A formally based parallelization of data mining algorithms for multi-core systems

We describe a novel, systematic approach to efficiently parallelizing data mining algorithms: starting with the representation of an algorithm as a sequential composition of functions, we formally transform it into a parallel form using higher-order functions for specifying parallelism. We implement the approach as an extension of the industrial-strength Java-based library Xelopes, and we illustrate its use by developing a multi-threaded Java program for the popular naive Bayes classification algorithm. In comparison with the popular MapReduce programming model, our resulting programs enable not only data-parallel, but also task-parallel implementation and a combination of both. Our experiments demonstrate an efficient parallelization and good scalability on multi-core processors.

A new MapReduce associative classifier based on a new storage format for large-scale imbalanced data

The process of knowledge discovery from big and high dimensional datasets has become a popular research topic. The classification problem is a key task in bioinformatics, business intelligence, decision science, astronomy, physics, etc. Building associative classifiers has been a notable research interest in recent years because of their superior accuracy. In associative classifiers, using under-sampling or over-sampling methods for imbalanced big datasets reduces accuracy or increases running time, respectively. Hence, there is a significant need to create efficient associative classifiers for imbalanced big data problems. These classifiers should be able to handle challenges such as memory usage, running time and efficiently exploring the search space. To this end, efficient calculation of measures is a primary objective for associative classifiers. In this paper, we propose a new efficient associative classifier for big imbalanced datasets. The proposed method is based on Rare-PEARs (a multi-objective evolutionary algorithm that efficiently discovers rare and reliable association rules) and is able to evaluate rules in a distributed manner by using a new storing data format. This format simplifies measures calculation and is fully compatible with the MapReduce programming model. We have applied the proposed method (RPII) on a well-known big dataset (ECBDL’14) and have compared our results with seven other learning methods. The experimental results show that RPII outperform other methods in sensitivity and final score measures (the values of sensitivity and final score measures were approximately 0.74 and 0.54 respectively). The results demonstrate that the proposed method is a good candidate for large-scale classification problems; furthermore, it achieves reasonable execution time when the target platform is a typical computer clusters.

Decoupled MapReduce for Shared-Memory Multi-Core Architectures

Modern multi-core processors exhibit high integration densities, e.g., up to several tens of cores. Multiple programming frameworks have emerged to facilitate the development of highly parallel applications. The MapReduce programming model, after having demonstrated its usability in the area of distributed computing systems, has been adapted to the needs of shared-memory multi-processors showing promising results in comparison with conventional multi-threaded libraries, e.g., pthreads. In this paper we enhance the traditional MapReduce architecture by decoupling the map and combine phases in order to boost parallel execution. We show that combiners’ memory intensive features limit the system’s degree of parallelism, thus resulting in sub-optimal hardware utilization, leaving space for further performance improvements. The proposed decoupled MapReduce architecture is evaluated into a NUMA server platform, showing that the adoption of the De-MapR runtime enables more efficient hardware utilization and competent run-time improvements. We demonstrate that the proposed solution achieves execution speedups of up to 2.46x compared to a state-of-the-art, shared-memory MapReduce library.

An Efficient Middle Layer Platform for Medical Imaging Archives.

Digital medical image usage is common in health services and clinics. These data have a vital importance for diagnosis and treatment; therefore, preservation, protection, and archiving of these data are a challenge. Rapidly growing file sizes differentiated data formats and increasing number of files constitute big data, which traditional systems do not have the capability to process and store these data. This study investigates an efficient middle layer platform based on Hadoop and MongoDB architecture using the state-of-the-art technologies in the literature. We have developed this system to improve the medical image compression method that we have developed before to create a middle layer platform that performs data compression and archiving operations. With this study, a platform using MapReduce programming model on Hadoop has been developed that can be scalable. MongoDB, a NoSQL database, has been used to satisfy performance requirements of the platform. A four-node Hadoop cluster has been built to evaluate the developed platform and execute distributed MapReduce algorithms. The actual patient medical images have been used to validate the performance of the platform. The processing of test images takes 15,599 seconds on a single node, but on the developed platform, this takes 8,153 seconds. Moreover, due to the medical imaging processing package used in the proposed method, the compression ratio values produced for the non-ROI image are between 92.12% and 97.84%. In conclusion, the proposed platform provides a cloud-based integrated solution to the medical image archiving problem.

Land forgery, black money rotation with genuine authentication using big data

Land forgery is predominant in India, the project aims to eliminate forgery activities while selling and buying the land with the guideline value. The money transaction made during the purchase of property majorly contributes to the black money problem in our country. To overcome this black money rotation and land forgery, both the Buyer and Seller has to register the land with their Aadhaar card to verify the genuineness of them. While verifying, if the system identifies any criminal cases filed against the buyer or seller or the land acquired by the seller is not authorized, it blocks the transaction. The process involves of document unique verification, legal activity verification and black money tracking, all the data are maintained in Hbase database and then integrated and stored in HDFS file storage. Map Reduce programming model is used to retrieve required information from big amount of data at a faster rate. If there are no such issues on the buyer and the seller. The overall transaction amount between the buyer and seller is finally referred to the Income Tax Department. If any illegal activity is detected in the transaction, the alert is sent to the Income Tax department and police officials so that the registration is blocked and black money rotation could be avoided.

An optimization framework for the capacity allocation and admission control of MapReduce jobs in cloud systems

Nowadays, we live in a Big Data world and many sectors of our economy are guided by data-driven decision processes. Big Data and Business Intelligence applications are facilitated by the MapReduce programming model, while, at infrastructural layer, cloud computing provides flexible and cost-effective solutions to provide on-demand large clusters. Capacity allocation in such systems, meant as the problem of providing computational power to support concurrent MapReduce applications in a cost-effective fashion, represents a challenge of paramount importance. In this paper we lay the foundation for a solution implementing admission control and capacity allocation for MapReduce jobs with a priori deadline guarantees. In particular, shared Hadoop 2.x clusters supporting batch and/or interactive jobs are targeted. We formulate a linear programming model able to minimize cloud resources costs and rejection penalties for the execution of jobs belonging to multiple classes with deadline guarantees. Scalability analyses demonstrated that the proposed method is able to determine the global optimal solution of the linear problem for systems including up to 10,000 classes in less than 1 s.

Sentiment analysis using semantic similarity and Hadoop MapReduce

Sentiment analysis or opinion mining is a domain that analyses people’s opinions, sentiments, evaluations, attitudes, and emotions from a written language; it had become a very active area of scientific research in recent years, especially with the development of social networks like Facebook and Twitter. In this paper we propose two new approaches to classify the tweets (look for the feeling expressed in the tweet), the first according to three classes : negative, positive or neutral, and the second according to two classes : negative or positive. Our first method consists in calculating the semantic similarity between the tweet to classify and three documents where each document represents a class (contains the words that represent a class); after the calculation of the similarity, the tweet takes the class of the document that has the greatest value of the semantic similarity with it. And the second method consists in calculating the semantic similarity between each word of the tweet to classify and the words “positive” and “negative” by proposing a new formula. We decide to do the analysis in a parallel and distributed way, using the Hadoop framework with the Hadoop distributed file system (HDFS) and the programming model MapReduce to solve the problem of the calculation time of the analysis if the dataset of the tweets is very large. The aim of our work is to combine between several domains, the information retrieval, semantic similarity, opinion mining or sentiment analysis and big data.

An analysis of MapReduce efficiency in document clustering using parallel K-means algorithm

One of the significant data mining techniques is clustering. Due to expansion and digitalization of each field, large datasets are being generated rapidly. Such large dataset clustering is a challenge for traditional sequential clustering algorithms due to huge processing time. Distributed parallel architectures and algorithms are thus helpful to achieve performance and scalability requirement of clustering large datasets. In this study, we design and experiment a parallel k-means algorithm using MapReduce programming model and compared the result with sequential k-means for clustering varying size of document dataset. The result demonstrates that proposed k-means obtains higher performance and outperformed sequential k-means while clustering documents.

Distributed nearest neighbor classification for large-scale multi-label data on spark

Modern data is characterized by its ever-increasing volume and complexity, particularly when data instances belong to many categories simultaneously. This learning paradigm is known as multi-label classification and one of its most renowned methods is the multi-label k nearest neighbor ( Ml-knn). The traditional implementations of this method are not feasible for large-scale multi-label data due to its complexity and memory restrictions. We propose a distributed Ml-knn implementation based on the MapReduce programming model, implemented on Apache Spark. We compare three strategies for distributed nearest neighbor search: 1) iteratively broadcasting instances, 2) using a distributed tree-based index structure, and 3) building hash tables to group instances. The experimental study evaluates the trade-off between the quality of the predictions and runtimes on 22 benchmark datasets, and compares the scalability using different sizes of data. The results indicate that the tree-based index strategy outperforms the other approaches, having a speedup of up to 266x for the largest dataset, while achieving an accuracy equivalent to the exact methods. This strategy enables Ml-knn to scale efficiently with respect to the size of the problem.

In-Mapper combiner based MapReduce algorithm for processing of big climate data

Big data refers to a collection of massive volume of data that cannot be processed by conventional data processing tools and technologies. In recent years, the data production sources are enlarged noticeably, such as high-end streaming devices, wireless sensor networks, satellite, wearable Internet of Things (IoT) devices. These data generation sources generate a massive volume of data in a continuous manner. The large volume of climate data is collected from the IoT weather sensor devices and NCEP. In this paper, the big data processing framework is proposed to integrate climate and health data and to find the correlation between the climate parameters and incidence of dengue. This framework is demonstrated with the help of MapReduce programming model, Hive, HBase and ArcGIS in a Hadoop Distributed File System (HDFS) environment. The following weather parameters such as minimum temperature, maximum temperature, wind, precipitation, solar and relative humidity are collected for the study are Tamil Nadu with the help of IoT weather sensor devices and NCEP. Proposed framework focuses only on climate data for 32 districts of Tamil Nadu where each district contains 1,57,680 rows and so there are 50,45,760 rows in total. Batch view precomputation for the monthly mean of various climate parameters would require 50,45,760 rows. Hence, this would create more latency in query processing. In order to overcome this issue, batch views can precompute for a smaller number of records and involve more computation to be done at query time. The In-Mapper based MapReduce framework is used to compute the monthly mean of climate parameter for each latitude and longitude. The experimental results prove the effectiveness of the response time for the In-Mapper based combiner algorithm is less when compared with the existing MapReduce algorithm.