MapReduce Approach Research Articles

SURVEY OF SIMILARITY JOIN ALGORITHMS BASED ON MAPREDUCE

Similarity Join is a data processing and analysis operation that retrieves all data pairs whose their distance is less than a pre-defined threshold. The similarity join algorithms are used in different real world applications such as finding similarity in documents, images, and strings. In this survey we will explain some of the similarity join algorithms which are based on MapReduce approach. These algorithms are: Set-Similarity Join, SSJ-2R, MRSimJoin, Pair-wise similarity, multi-sig-er method, Trie-join, and PreJoin algorithm. We then make a comparison between these algorithms according to some criteria and discuss the results. Article DOI: https://dx.doi.org/10.20319/mijst.2016.s21.214234 This work is licensed under the Creative Commons Attribution-Non-commercial 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

Large-scale virtual screening on public cloud resources with Apache Spark

BackgroundStructure-based virtual screening is an in-silico method to screen a target receptor against a virtual molecular library. Applying docking-based screening to large molecular libraries can be computationally expensive, however it constitutes a trivially parallelizable task. Most of the available parallel implementations are based on message passing interface, relying on low failure rate hardware and fast network connection. Google’s MapReduce revolutionized large-scale analysis, enabling the processing of massive datasets on commodity hardware and cloud resources, providing transparent scalability and fault tolerance at the software level. Open source implementations of MapReduce include Apache Hadoop and the more recent Apache Spark.ResultsWe developed a method to run existing docking-based screening software on distributed cloud resources, utilizing the MapReduce approach. We benchmarked our method, which is implemented in Apache Spark, docking a publicly available target receptor against sim 2.2 M compounds. The performance experiments show a good parallel efficiency (87%) when running in a public cloud environment.ConclusionOur method enables parallel Structure-based virtual screening on public cloud resources or commodity computer clusters. The degree of scalability that we achieve allows for trying out our method on relatively small libraries first and then to scale to larger libraries. Our implementation is named Spark-VS and it is freely available as open source from GitHub (https://github.com/mcapuccini/spark-vs).Graphical abstract.

Scalable subgraph enumeration in MapReduce: a cost-oriented approach

Subgraph enumeration, which aims to find all the subgraphs of a large data graph that are isomorphic to a given pattern graph, is a fundamental graph problem with a wide range of applications. However, existing sequential algorithms for subgraph enumeration fall short in handling large graphs due to the involvement of computationally intensive subgraph isomorphism operations. Thus, some recent researches focus on solving the problem using MapReduce. Nevertheless, exiting MapReduce approaches are not scalable to handle very large graphs since they either produce a huge number of partial results or consume a large amount of memory. Motivated by this, in this paper, we propose a new algorithm $$\mathsf {Twin}$$Twin$$\mathsf {Twig}$$Twig$$\mathsf {Join}$$Join based on a left-deep-join framework in MapReduce, in which the basic join unit is a $$\mathsf {Twin}$$Twin$$\mathsf {Twig}$$Twig (an edge or two incident edges of a node). We show that in the Erdos---Renyi random graph model, $$\mathsf {Twin}$$Twin$$\mathsf {Twig}$$Twig$$\mathsf {Join}$$Join is instance optimal in the left-deep-join framework under reasonable assumptions, and we devise an algorithm to compute the optimal join plan. We further discuss how our approach can be adapted to handle the power-law random graph model. Three optimization strategies are explored to improve our algorithm. Ultimately, by aggregating equivalent nodes into a compressed node, we construct the compressed graph, upon which the subgraph enumeration is further improved. We conduct extensive performance studies in several real graphs, one of which contains billions of edges. Our approach significantly outperforms existing solutions in all tests.

Hybrid of anomaly-based and specification-based IDS for Internet of Things using unsupervised OPF based on MapReduce approach

Internet of Things (IoT) is a novel paradigm in computer networks in which resource-constrained objects connect to unreliable Internet by using a wide range of technologies. The insecure nature of the Internet and wireless sensor networks, that are the main components of IoT, make IoT vulnerable to different attacks, especially routing attacks (as insider attacks). A novel real-time hybrid intrusion detection framework is proposed in this study that consists of anomaly-based and specification-based intrusion detection modules for detecting two well-known routing attacks in IoT called sinkhole and selective-forwarding attacks. For this purpose, the specification-based intrusion detection agents, that are located in the router nodes, analyze the behavior of their host nodes and send their local results to the root node through normal data packets. In addition, an anomaly-based intrusion detection agent, that is located in the root node, employs the unsupervised optimum-path forest algorithm for projecting clustering models by using incoming data packets. This agent, which is based on the MapReduce architecture, can work in a distributed platform for projecting clustering models and consequently parallel detecting of anomalies as a global detection approach. The proposed method makes decision about suspicious behavior by using a voting mechanism. Notably, the proposed method is also extended to detect wormhole attack. The deployment of the hybrid proposed model is investigated in a smart-city scenario by an existing platform, as well. The free network's scale and the ability to identify malicious nodes are two key features of the proposed framework that are evaluated through different experiments in this study. The experimental results of simulated scenarios showed that the proposed hybrid method can achieve true positive rate of 76.19% and false positive rate of 5.92% when both sinkhole and selective-forwarding attacks were launched simultaneously. These rates in detecting wormhole attack are 96.02% and 2.08%, respectively.

A survey paper on logical perspective to manage BigData with incremental map reduce

Big Data is a collection of a huge and complex data that it becomes extremely drab to seize, store, process, reclaim and inspect it with the help of on hand database management tools or traditional data processing techniques. Now a day, data is constantly evolving and becomes a big data. The data is being generated from different sources - undertaking, social media, sensors, digital images, video, audio and clickstreams for domains together with healthcare, retail, energy and utilities. It is intended to scale up from single server to thousands of machines, each offering local computations and storage Big data with 3 Vs: volume, variety and velocity. For processing such big volume of data, variety of data and the data with high velocity and having high storage capacity, we introduced Hadoop which is evolved day by day. We used MapReduce at this point as a programming model. In this paper we worn Incremental MapReduce most extensively used framework for processing big data. To improve the time of processing big data and optimizing data content of big data we applied PageRank and k-means iteratively along with MapReduce. Therefore to process big data incremental MapReduce approach is used. Incremental MapReduce 1) performs key-value pair level incremental processing, 2) supports complicated duplication computation, which is widely used in data mining applications. That means incremental MapReduce processes big data in a less time and stores it in a more optimized form.

Big Data Analytics to Predict Breast Cancer Recurrence on SEER Dataset using MapReduce Approach

traditional data analytic might not have the capacity to handle enormous amount of data. Due to the rapid growth of information, solutions need to be contemplated and provided in order to handle and extract value and knowledge from these data sets. Moreover, decision makers should have the capacity to increase significant bits of knowledge from such fluctuated and quickly evolving information. Such esteem can be given utilizing big data analytic, which is the utilization of advanced analytic techniques on big data using MapReduce approach. This paper examines to develop a high performance platform to efficiently analyse big SEER (Surveillance, Epidemiology, and End Results) breast cancer data set using MapReduce to find the recurrence of breast cancer. Keywordscancer; Big data, Classification; Data analytics, MapReduce, SEER.

Parallel Computing to Predict Breast Cancer Recurrence on SEER Dataset using Map-Reduce Approach

to the late overpowering development rate of large scale data, the advancement of handling faster processing algorithms with optimal execution has turned into a critical need of the time. In this paper, parallel Map-Reduce algorithm is proposed, that encourages concurrent participation of various computing hubs to develop a classifier on SEER breast cancer data set. Our algorithm can prompt supported models whose speculation execution is near the respective baseline classifier. By exploiting their own parallel architecture the algorithm increases noteworthy speedup. In addition, the algorithm don't require singular processing hubs to communicate with each other, to share their data or to share the knowledge got from their data and consequently, they are powerful in safeguarding privacy of computation also. This paper utilized the Map-Reduce framework to implement the algorithms and experimented onSEER breast cancer data sets to exhibit the execution as far as classification accuracy and speedup. Keywordscancer; Big dataanalytics, Classification; Parallel Computing; MapReduce, SEER.

Fuzzy rule based classification systems for big data with MapReduce: granularity analysis

Due to the vast amount of information available nowadays, and the advantages related to the processing of this data, the topics of big data and data science have acquired a great importance in the current research. Big data applications are mainly about scalability, which can be achieved via the MapReduce programming model.It is designed to divide the data into several chunks or groups that are processed in parallel, and whose result is "assembled" to provide a single solution. Among different classification paradigms adapted to this new framework, fuzzy rule based classification systems have shown interesting results with a MapReduce approach for big data. It is well known that the performance of these types of systems has a strong dependence on the selection of a good granularity level for the Data Base. However, in the context of MapReduce this parameter is even harder to determine as it can be also related with the number of Maps chosen for the processing stage. In this paper, we aim at analyzing the interrelation between the number of labels of the fuzzy variables and the scarcity of the data due to the data sampling in MapReduce. Specifically, we consider that as the partitioning of the initial instance set grows, the level of granularity necessary to achieve a good performance also becomes higher. The experimental results, carried out for several Big Data problems, and using the Chi-FRBCS-BigData algorithms, support our claims.

Scalable Process Discovery Using Map-Reduce

Process discovery is an approach to extract process models from event logs. Given the distributed nature of modern information systems, event logs are likely to be distributed across different physical machines. Map-Reduce is a scalable approach for efficient computations on distributed data. In this paper we present Map-Reduce implementations of two well-known process mining algorithms to take advantage of the scalability of the Map-Reduce approach. We present the design of a series of mappers and reducers to compute the log-based ordering relations from distributed event logs. These can then be used to discover a process model. We provide experimental results that show the performance and scalability of our implementations.

MRPack: Multi-Algorithm Execution Using Compute-Intensive Approach in MapReduce.

Large quantities of data have been generated from multiple sources at exponential rates in the last few years. These data are generated at high velocity as real time and streaming data in variety of formats. These characteristics give rise to challenges in its modeling, computation, and processing. Hadoop MapReduce (MR) is a well known data-intensive distributed processing framework using the distributed file system (DFS) for Big Data. Current implementations of MR only support execution of a single algorithm in the entire Hadoop cluster. In this paper, we propose MapReducePack (MRPack), a variation of MR that supports execution of a set of related algorithms in a single MR job. We exploit the computational capability of a cluster by increasing the compute-intensiveness of MapReduce while maintaining its data-intensive approach. It uses the available computing resources by dynamically managing the task assignment and intermediate data. Intermediate data from multiple algorithms are managed using multi-key and skew mitigation strategies. The performance study of the proposed system shows that it is time, I/O, and memory efficient compared to the default MapReduce. The proposed approach reduces the execution time by 200% with an approximate 50% decrease in I/O cost. Complexity and qualitative results analysis shows significant performance improvement.

Applying Mapreduce and Network Traffic Analysis to Control Access to Information Resources

обработка информации в настоящее время является одной из наиболее актуальных задач. С ростом и развитием информационных и телекоммуникационных технологий выросли и объемы передаваемой информации по сети Интернет. одновременно с обработкой информации встает вопрос ее защиты. Предложен подход к построению распределенной вычислительной системы, осуществляющей обработку снимков сетевого трафика за приемлемое время и обеспечивающей близкий к линейному рост производительности при наращивании вычислительных мощностей. АНАлИЗ СеТеВоГо ТРАФИКА; больШИе дАННые; MAPREDUCE; HADOOP. Nowadays information security is an important issue. Network traffic analysis is widely used by Internet Service Providers to evaluate network performance, to collect statistics and to detect vulnerabilities. To analyze traffic traces collected from a large network it is required a computer system where both storage and computing resources can be easily scaled out to handle and process multi-Terabyte files. Cloud computing platforms and cluster file systems could provide resizable compute and storage capacity. The MapReduce programming model developed by Google in 2004 allows processing huge amounts of data in distributed manner by defining the map and reduce functions. The given paper proposes a cloud-computing framework based on a MapReduce approach for fast internet traffic analytics. DEEP PACKET INSPECTION; BIGDATA; MAPREDUCE; HADOOP.

Scalable subgraph enumeration in MapReduce

Subgraph enumeration, which aims to find all the subgraphs of a large data graph that are isomorphic to a given pattern graph, is a fundamental graph problem with a wide range of applications. However, existing sequential algorithms for subgraph enumeration fall short in handling large graphs due to the involvement of computationally intensive subgraph isomorphism operations. Thus, some recent researches focus on solving the problem using MapReduce. Nevertheless, exiting MapReduce approaches are not scalable to handle very large graphs since they either produce a huge number of partial results or consume a large amount of memory. Motivated by this, in this paper, we propose a new algorithm TwinTwigJoin based on a left-deep-join framework in MapReduce, in which the basic join unit is a TwinTwig (an edge or two incident edges of a node). We show that in the Erdös-Rényi random-graph model, TwinTwigJoin is instance optimal in the left-deep-join framework under reasonable assumptions, and we devise an algorithm to compute the optimal join plan. Three optimization strategies are explored to improve our algorithm. Furthermore, we discuss how our approach can be adapted in the power-law random-graph model. We conduct extensive performance studies in several real graphs, one of which contains billions of edges. Our approach significantly outperforms existing solutions in all tests.

ROSEFW-RF: The winner algorithm for the ECBDL’14 big data competition: An extremely imbalanced big data bioinformatics problem

The application of data mining and machine learning techniques to biological and biomedicine data continues to be an ubiquitous research theme in current bioinformatics. The rapid advances in biotechnology are allowing us to obtain and store large quantities of data about cells, proteins, genes, etc., that should be processed. Moreover, in many of these problems such as contact map prediction, the problem tackled in this paper, it is difficult to collect representative positive examples. Learning under these circumstances, known as imbalanced big data classification, may not be straightforward for most of the standard machine learning methods.In this work we describe the methodology that won the ECBDL’14 big data challenge for a bioinformatics big data problem. This algorithm, named as ROSEFW-RF, is based on several MapReduce approaches to (1) balance the classes distribution through random oversampling, (2) detect the most relevant features via an evolutionary feature weighting process and a threshold to choose them, (3) build an appropriate Random Forest model from the pre-processed data and finally (4) classify the test data. Across the paper, we detail and analyze the decisions made during the competition showing an extensive experimental study that characterize the way of working of our methodology. From this analysis we can conclude that this approach is very suitable to tackle large-scale bioinformatics classifications problems.

Leveraging big-data for business process analytics

Purpose – This paper aims to present a solution that enables organizations to monitor and analyse the performance of their business processes by means of Big Data technology. Business process improvement can drastically influence in the profit of corporations and helps them to remain viable. However, the use of traditional Business Intelligence systems is not sufficient to meet today ' s business needs. They normally are business domain-specific and have not been sufficiently process-aware to support the needs of process improvement-type activities, especially on large and complex supply chains, where it entails integrating, monitoring and analysing a vast amount of dispersed event logs, with no structure, and produced on a variety of heterogeneous environments. This paper tackles this variability by devising different Big-Data-based approaches that aim to gain visibility into process performance. Design/methodology/approach – Authors present a cloud-based solution that leverages (BD) technology to provide essential insights into business process improvement. The proposed solution is aimed at measuring and improving overall business performance, especially in very large and complex cross-organisational business processes, where this type of visibility is hard to achieve across heterogeneous systems. Findings – Three different (BD) approaches have been undertaken based on Hadoop and HBase. We introduced first, a map-reduce approach that it is suitable for batch processing and presents a very high scalability. Secondly, we have described an alternative solution by integrating the proposed system with Impala. This approach has significant improvements in respect with map reduce as it is focused on performing real-time queries over HBase. Finally, the use of secondary indexes has been also proposed with the aim of enabling immediate access to event instances for correlation in detriment of high duplication storage and synchronization issues. This approach has produced remarkable results in two real functional environments presented in the paper. Originality/value – The value of the contribution relies on the comparison and integration of software packages towards an integrated solution that is aimed to be adopted by industry. Apart from that, in this paper, authors illustrate the deployment of the architecture in two different settings.

Evolutionary Feature Selection for Big Data Classification: A MapReduce Approach

Nowadays, many disciplines have to deal with big datasets that additionally involve a high number of features. Feature selection methods aim at eliminating noisy, redundant, or irrelevant features that may deteriorate the classification performance. However, traditional methods lack enough scalability to cope with datasets of millions of instances and extract successful results in a delimited time. This paper presents a feature selection algorithm based on evolutionary computation that uses the MapReduce paradigm to obtain subsets of features from big datasets. The algorithm decomposes the original dataset in blocks of instances to learn from them in the map phase; then, the reduce phase merges the obtained partial results into a final vector of feature weights, which allows a flexible application of the feature selection procedure using a threshold to determine the selected subset of features. The feature selection method is evaluated by using three well-known classifiers (SVM, Logistic Regression, and Naive Bayes) implemented within the Spark framework to address big data problems. In the experiments, datasets up to 67 millions of instances and up to 2000 attributes have been managed, showing that this is a suitable framework to perform evolutionary feature selection, improving both the classification accuracy and its runtime when dealing with big data problems.

A MapReduce Approach to Address Big Data Classification Problems Based on the Fusion of Linguistic Fuzzy Rules

AbstractThe big data term is used to describe the exponential data growth that has recently occurred and represents an immense challenge for traditional learning techniques. To deal with big data classification problems we propose the Chi-FRBCS-BigData algorithm, a linguistic fuzzy rule-based classification system that uses the MapReduce framework to learn and fuse rule bases. It has been developed in two versions with different fusion processes. An experimental study is carried out and the results obtained show that the proposal is able to handle these problems providing competitive results.

Pervasive Data Processing

Abstract This paper presents a data processing framework for enabling MapReduce approach to be available in pervasive networks, including sensor networks and Internet of Things (IoT). It is unique among other existing MapReduce-based approaches, because it can locally process data maintained on nodes in pervasive networks. It dynamically deploys programs for data processing at the nodes that have the target data as a map step and executes the programs with the local data. Finally, it aggregates the results of the programs to certain nodes as a reduce step. The paper proposes the architecture of the framework and describes its basic performance and application.

Data Mining Techniques in Parallel Environment- A Comprehensive Survey

Data mining is the process of discovering interesting and useful patterns and relationships in large volumes of data. The valuable knowledge can be discovered through the process of data mining for the further use and prediction. We have different data mining techniques like clustering classification and association. Classification is one of the major techniques to discover the patterns in huge amount of data. This technique is widely used in many fields. We have a large volume of data and if we extract the data sequentially then it will take a lot of timing. So if we extract the data parallely, the amount of time taken can be reduced. We can use parallel techniques when there is a large volume of data and we want to extract the data in very few seconds. We can implement this techniques using different approaches like MPI, OPENMP, using CUDA or using Map Reduce approach. Here in this paper we will discuss data mining techniques classification by decision tree induction and knearest neighbors using both sequential approach as well as parallel approach. General Terms Data mining, Classification

Scaling up MapReduce-based Big Data Processing on Multi-GPU systems

MapReduce is a popular data-parallel processing model encompassed with recent advances in computing technology and has been widely exploited for large-scale data analysis. The high demand on MapReduce has stimulated the investigation of MapReduce implementations with different architectural models and computing paradigms, such as multi-core clusters, Clouds, Cubieboards and GPUs. Particularly, current GPU-based MapReduce approaches mainly focus on single-GPU algorithms and cannot handle large data sets, due to the limited GPU memory capacity. Based on the previous multi-GPU MapReduce version MGMR, this paper proposes an upgrade version MGMR++ to eliminate GPU memory limitation and a pipelined version, PMGMR, to handle the Big Data challenge through both CPU memory and hard disks. MGMR++ is extended from MGMR with flexible C++ templates and CPU memory utilization, while PMGMR fine-tuned the performance through the latest GPU features such as streams and Hyper-Q as well as hard disk utilization. Compared to MGMR (Jiang et al., Cluster Computing 2013), the proposed schemes achieve about 2.5-fold performance improvement, increase system scalability, and allow programmers to write straightforward MapReduce code for Big Data.

A MapReduce Technique to Mosaic Continental-Scale Weather Radar Data in Real-Time

It is necessary to create continental-scale mosaics of radar data in real-time for applications ranging from precipitation estimation, hail diagnosis and tornado warnings to public awareness. Because of the high temporal and spatial resolution of data available from the United States' network of weather radars, creating radar mosaics in real-time has been possible only through compromises on the quality, timeliness or resolution of the mosaics. MapReduce is a programming model that can be employed for processing and generating large data sets by distributing parallel computations and data storage across a distributed cluster of machines. In this paper, a MapReduce approach to computing radar mosaics on a distributed cluster of compute nodes is presented. The approach is massively scalable, and is able to create high-resolution 3D radar mosaics over the Continental United States in real-time.