MapReduce Approach Research Articles

Counting Triangles in Massive Graphs with MapReduce

Graphs and networks are used to model interactions in a variety of contexts. There is a growing need to quickly assess the characteristics of a graph in order to understand its underlying structure. Some of the most useful metrics are triangle-based and give a measure of the connectedness of mutual friends. This is often summarized in terms of clustering coefficients, which measure the likelihood that two neighbors of a node are themselves connected. Computing these measures exactly for large-scale networks is prohibitively expensive in both memory and time. However, a recent wedge sampling algorithm has proved successful in efficiently and accurately estimating clustering coefficients. In this paper, we describe how to implement this approach in MapReduce to deal with massive graphs. We show results on publicly-available networks, the largest of which is 132M nodes and 4.7B edges, as well as artificially generated networks (using the Graph500 benchmark), the largest of which has 240M nodes and 8.5B edges. We can estimate the clustering coefficient by degree bin (e.g., we use exponential binning) and the number of triangles per bin, as well as the global clustering coefficient and total number of triangles, in an average of 0.33 seconds per million edges plus overhead (approximately 225 seconds total for our configuration). The technique can also be used to study triangle statistics such as the ratio of the highest and lowest degree, and we highlight differences between social and non-social networks. To the best of our knowledge, these are the largest triangle-based graph computations published to date.

Market Basket Analysis algorithms with MapReduce

The MapReduce approach has been popular in computing large scale data since Google implemented its platform on Google Distributed File Systems (GFS) followed by Amazon Web Service (AWS) providing the Apache Hadoop platform in inexpensive computing nodes. Map/Reduce motivates to redesign and convert the existing sequential algorithms to MapReduce as restricted parallel programming so that the paper proposes Market Basket Analysis algorithm with MapReduce as well as apriority property. Two algorithms are proposed by adapting an existing Apriori‐algorithm and building a simple algorithm that sorts data sets and converts it to (key, value) pairs to fit with MapReduce. It is executed on Amazon EC2 Map/Reduce platform. The experimental results show that the Apriori‐algorithm does not perform as well as the simple algorithm. Using the simple algorithm, the code with Map/Reduce increases the performance by adding more nodes, but at a certain point there is a bottleneck that does not allow further performance gain. It is believed that the operations of distributing, aggregating, and reducing data in Map/Reduce, cause the bottleneck. WIREs Data Mining Knowl Discov 2013, 3:445–452. doi: 10.1002/widm.1107This article is categorized under: Algorithmic Development > Association Rules Application Areas > Business and Industry Fundamental Concepts of Data and Knowledge > Big Data Mining

A Practical Approach for Scalable Record Linkage on Hadoop

As increasing data have been collected in many applications, we have to face with millions of data in record linkage. With respect to traditional methods, there comes out a big challenge in performance while dealing with massive data. Parallel computing framework, such as MapReduce, has become an efficient and practical way to address this problem. In this paper, we propose a practical 3-phase MapReduce approach that fulfills blocking, filtering, and linking in 3 consecutive processes on Hadoop cluster. Experiments show that our approach functions efficiently and effectively with keeping high recall in contrast to tradition method.

A Comparative Study for Optimization of Video File Compression in Cloud Environment

Many organizations like hospitals for telemedicine, journalism for live-telecast and academias are using a service video-on-demand for delivering the lectures and research contents to the remote locations across the globe. The videos to be broadcasted are time and resource consuming due to the large amount of data and due to these constraints, for getting fast access over Internet and mobile devices, such video applications need to be compressed into another format. The usage of videos is occasional so to save huge infrastructure cost and time, the Infrastructure as a Service (IaaS) Cloud systems can be leveraged. In this paper, an attempt has been made to design, implement and optimize the performance of Digital Video to MPEG4 transcoding in the Cloud environment using Meghdoot (an Open-Source Cloud stack). The classical MapReduce approach is used to rationalize the use of resources by exploring on demand computing and performs parallel video conversion thereby reducing the video encoding times. Experimental results point out to suitability of better performance that by varying the technique of splitting the video file size of fragments that is through Mencoder and through default Hadoop Splitting. The comparison of both the systems to get the best compression times will help us to optimize the Cloud resources that further helps in trade-off between time, cost and quality.

MapReduce approach to relational influence propagation in complex networks

The relational label propagation problem for large data sets using MapReduce programming model was considered in the paper. The method we propose estimates class probability in relational domain in the networks. The method was examined on large real telecommunication data set. The results indicated that it could be used successfully to classify networks’ nodes and, thanks to that, new offerings or tariffs might be proposed to customers who belong to other providers. Moreover, basic properties of relational label propagation were examined and reported.

Relational large scale multi-label classification method for video categorization

The problem of automated video categorization in large datasets is considered in the paper. A new Iterative Multi-label Propagation (IMP) algorithm for relational learning in multi-label data is proposed. Based on the information of the already categorized videos and their relations to other videos, the system assigns suitable categories—multiple labels to the unknown videos. The MapReduce approach to the IMP algorithm described in the paper enables processing of large datasets in parallel computing. The experiments carried out on 5-million videos dataset revealed the good efficiency of the multi-label classification for videos categorization. They have additionally shown that classification of all unknown videos required only several parallel iterations.

A scalable and accurate method for classifying protein–ligand binding geometries using a MapReduce approach

We present a scalable and accurate method for classifying protein–ligand binding geometries in molecular docking. Our method is a three-step process: the first step encodes the geometry of a three-dimensional (3D) ligand conformation into a single 3D point in the space; the second step builds an octree by assigning an octant identifier to every single point in the space under consideration; and the third step performs an octree-based clustering on the reduced conformation space and identifies the most dense octant. We adapt our method for MapReduce and implement it in Hadoop. The load-balancing, fault-tolerance, and scalability in MapReduce allow screening of very large conformation spaces not approachable with traditional clustering methods. We analyze results for docking trials for 23 protein–ligand complexes for HIV protease, 21 protein–ligand complexes for Trypsin, and 12 protein–ligand complexes for P38alpha kinase. We also analyze cross docking trials for 24 ligands, each docking into 24 protein conformations of the HIV protease, and receptor ensemble docking trials for 24 ligands, each docking in a pool of HIV protease receptors. Our method demonstrates significant improvement over energy-only scoring for the accurate identification of native ligand geometries in all these docking assessments. The advantages of our clustering approach make it attractive for complex applications in real-world drug design efforts. We demonstrate that our method is particularly useful for clustering docking results using a minimal ensemble of representative protein conformational states (receptor ensemble docking), which is now a common strategy to address protein flexibility in molecular docking.

Cloud MapReduce for Monte Carlo bootstrap applied to Metabolic Flux Analysis

The MapReduce architectural pattern popularized by Google has successfully been utilized in several scientific applications. Up until now, MapReduce is rarely employed in the field of Systems Biology. We investigate whether a MapReduce approach utilizing on-demand resources from a Cloud is suitable to perform simulation tasks in the area of Metabolic Flux Analysis (MFA). An Amazon ElasticMapReduce Cloud implementation of the parallel, parametric Monte Carlo bootstrap in the context to 13C-MFA is presented. The seamless integration of the application into a service-oriented, BPEL-based scientific workflow framework is shown. A comparison of a straightforward MapReduce implementation using the Hadoop streaming interface on various Amazon ElasticMapReduce instance types and a single CPU core computation approach reveals a speedup of 17 on 64 Amazon cores. I/O operations on many small files within the Reduce step were identified as the limiting step. By exploiting the Hadoop Java API, making use of built-in data types and tuning problem-specific Hadoop parameters, the I/O issues could be resolved. With the revised implementation, a speedup of up to 48 could be achieved on 64 Amazon cores. To investigate the runtimes of a realistic 13C-MFA analysis, 50,000 Monte Carlo samples with a typical metabolic network model have been performed on 20 virtual nodes in 24 h and 23 min with a total cost of $384. Our work demonstrates the possibility to perform scalable Systems Biology applications using Amazon’s Cloud MapReduce service.

Data and task parallelism in ILP using MapReduce

Nearly two decades of research in the area of Inductive Logic Programming (ILP) have seen steady progress in clarifying its theoretical foundations and regular demonstrations of its applicability to complex problems in very diverse domains. These results are necessary, but not sufficient, for ILP to be adopted as a tool for data analysis in an era of very large machine-generated scientific and industrial datasets, accompanied by programs that provide ready access to complex relational information in machine-readable forms (ontologies, parsers, and so on). Besides the usual issues about the ease of use, ILP is now confronted with questions of implementation. We are concerned here with two of these, namely: can an ILP system construct models efficiently when (a) Dataset sizes are too large to fit in the memory of a single machine; and (b) Search space sizes becomes prohibitively large to explore using a single machine. In this paper, we examine the applicability to ILP of a popular distributed computing approach that provides a uniform way for performing data and task parallel computations in ILP. The MapReduce programming model allows, in principle, very large numbers of processors to be used without any special understanding of the underlying hardware or software involved. Specifically, we show how the MapReduce approach can be used to perform the coverage-test that is at the heart of many ILP systems, and to perform multiple searches required by a greedy set-covering algorithm used by some popular ILP systems. Our principal findings with synthetic and real-world datasets for both data and task parallelism are these: (a) Ignoring overheads, the time to perform the computations concurrently increases with the size of the dataset for data parallelism and with the size of the search space for task parallelism. For data parallelism this increase is roughly in proportion to increases in dataset size; (b) If a MapReduce implementation is used as part of an ILP system, then benefits for data parallelism can only be expected above some minimal dataset size, and for task parallelism can only be expected above some minimal search-space size; and (c) The MapReduce approach appears better suited to exploit data-parallelism in ILP.

MapReduce Platform for Parallel Machine Learning on Large-scale Dataset

We propose a computing platform for parallel machine learning. Learning from large-scale data has become common, so that parallelization techniques are increasingly applied to machine learning algorithms in order to reduce calculation time. Problems of parallelization are implementation costs and calculation overheads. Firstly, we formulate MapReduce programming model specialized in parallel machine learning. It represents learning algorithms as iterations of following two phases: applying data to machine learning models and updating model parameters. This model is able to describe various kinds of machine learning algorithms, such as k-means clustering, EM algorithm, and linear SVM, with comparable implementation cost to the original MapReduce. Secondly, we propose a fast machine learning platform which reduces the processing overheads at iterative procedures of machine learning. Machine learning algorithms iteratively read the same training data in the data application phase. Our platform keeps the training data in local memories of each worker during iterative procedures, which leads to acceleration of data access. We evaluate performance of our platform on three experiments. Our platform executes k-means clustering 2.85 to 118 times faster than the MapReduce approach, and shows 9.51 times speedup with 40 processing cores against 8 cores. We also show the performance of Variational Bayes clustering and linear SVM implemented on our platform.

Implementing and Optimizing Multiple Group by Query in a MapReduce Approach

MapReduce model is a new parallel programming model initially developed for large-scale web content processing. Data analysis meets the issue of how to do calculation over extremely large dataset. The arrival of MapReduce provides a chance to utilize commodity hardware for massively parallel data analysis applications. The translation and optimization from relational algebra operators to MapReduce programs is still an open and dynamic research field. In this paper, we focus on a special type of data analysis query, namely, multiple group by query. We first study the communication cost of MapReduce model, then we give an initial implementation of multiple group by query. We then propose an optimized version which addresses and improves the communication cost issues. Our optimized version shows a better accelerating ability and a better scalability than the other version.

Google’s MapReduce programming model — Revisited

Google’s MapReduce programming model serves for processing large data sets in a massively parallel manner. We deliver the first rigorous description of the model including its advancement as Google’s domain-specific language Sawzall. To this end, we reverse-engineer the seminal papers on MapReduce and Sawzall, and we capture our findings as an executable specification. We also identify and resolve some obscurities in the informal presentation given in the seminal papers. We use typed functional programming (specifically Haskell) as a tool for design recovery and executable specification. Our development comprises three components: (i) the basic program skeleton that underlies MapReduce computations; (ii) the opportunities for parallelism in executing MapReduce computations; (iii) the fundamental characteristics of Sawzall’s aggregators as an advancement of the MapReduce approach. Our development does not formalize the more implementational aspects of an actual, distributed execution of MapReduce computations.

AMPICILLIN BY INJECTION AND BY MOUTH IN THE TREATMENT OF ACUTE GONORRHOEA.

<h3>Abstract</h3> <h3>Background</h3> <i>De novo</i> transcriptome assembly is an important technique for understanding gene expression in non-model organisms. Many de novo assemblers using the de Bruijn graph of a set of the RNA sequences rely on in-memory representation of this graph. However, current methods analyse the complete set of read-derived k-mer sequence at once, resulting in the need for computer hardware with large shared memory. <h3>Results</h3> We introduce a novel approach that clusters k-mers as the first step. The clusters correspond to small sets of gene products, which can be processed quickly to give candidate transcripts. We implement the clustering step using the MapReduce approach for parallelising the analysis of large datasets, which enables the use of compute clusters. The computational task is distributed across the compute system, and no specialised hardware is required. Using this approach, we have re-implemented the Inchworm module from the widely used Trinity pipeline, and tested the method in the context of the full Trinity pipeline. Validation tests on a range of real datasets show large reductions in the runtime and per-node memory requirements, when making use of a compute cluster. <h3>Conclusions</h3> Our study shows that MapReduce-based clustering has great potential for distributing challenging sequencing problems, without loss of accuracy. Although we have focussed on the Trinity package, we propose that such clustering is a useful initial step for other assembly pipelines.