MapReduce Version Research Articles

Tracking Data Provenance of Archaeological Temporal Information in Presence of Uncertainty

The interpretation process is one of the main tasks performed by archaeologists who, starting from ground data about evidences and findings, incrementally derive knowledge about ancient objects or events. Very often more than one archaeologist contributes in different time instants to discover details about the same finding and thus, it is important to keep track of history and provenance of the overall knowledge discovery process. To this aim, we propose a model and a set of derivation rules for tracking and refining data provenance during the archaeological interpretation process. In particular, among all the possible interpretation activities, we concentrate on the one concerning the dating that archaeologists perform to assign one or more time intervals to a finding to define its lifespan on the temporal axis. In this context, we propose a framework to represent and derive updated provenance data about temporal information after the mentioned derivation process. Archaeological data, and in particular their temporal dimension, are typically vague, since many different interpretations can coexist, thus, we will use Fuzzy Logic to assign a degree of confidence to values and Fuzzy Temporal Constraint Networks to model relationships between dating of different findings represented as a graph-based dataset. The derivation rules used to infer more precise temporal intervals are enriched to manage also provenance information and their following updates after a derivation step. A MapReduce version of the path consistency algorithm is also proposed to improve the efficiency of the refining process on big graph-based datasets.

Evaluating an Application Aware Distributed Dijkstra Shortest Path Algorithm in Hybrid Cloud/Edge Environments

To increase the flexibility and the dynamism of communication networks, Software Defined Networking (SDN) has emerged as a challenging approach to decouple control and data planes, using a logically centralized controller able to manage the underlying network resources. However, traditional network solutions can not be always used in SDN. In this paper, we deal with routing issues in the setup of dynamic SDN spanning Fog/Edge and IoT systems for supporting the new generation of applications. In particular, we present a modified version of Dijkstra's routing algorithm that can optimize complex routing metrics and uses MapReduce to speed up the configuration of routers in a software-defined network. The system can optimize the packet routing accordingly to different parameters including, e.g., hops, latency, and energy efficiency policies. To show the effective benefits of the proposed solution, we performed evaluations on the revised MapReduce version of the Dijkstra routing algorithm considering a highly scalable network topology with thousands of virtual nodes.

Efficient parallel derivation of short distinguishing sequences for nondeterministic finite state machines using MapReduce

Distinguishing sequences are widely used in finite state machine-based conformance testing to solve the state identification problem. In this paper, we address the scalability issue encountered while deriving distinguishing sequences from complete observable nondeterministic finite state machines by introducing a massively parallel MapReduce version of the well-known Exact Algorithm. To the best of our knowledge, this is the first study to tackle this task using the MapReduce approach. First, we give a concise overview of the well-known Exact Algorithm for deriving distinguishing sequences from nondeterministic finite state machines. Second, we propose a parallel algorithm for this problem using the MapReduce approach and analyze its communication cost using Afrati et al. model. Furthermore, we conduct a variety of intensive and comparative experiments on a wide range of finite state machine classes to demonstrate that our proposed solution is efficient and scalable.

MR-OVnTSA: a heuristics based sensitive pattern hiding approach for big data

This paper presents a novel ‘MapReduce Based Optimum Victim Item and Transaction Selection Approach (MR-OVnTSA)’ that provides a feasible and intelligent solution for protecting sensitive frequent itemsets present in big data. The approach advocates to resolve the captious challenges, existing knowledge hiding algorithms are encountering. The proposed solution optimally minimizes the side effect of hiding process on non-sensitive information, and maintains a balance between knowledge and privacy as well as handles the exponential growth in data volume efficiently. The algorithm plugs the most optimum item and transaction as victim, by intelligently analyzing their coverage value i.e. it chooses one with maximal impact on sensitive knowledge but minimal on non-sensitive information. Further, the MapReduce version of the proposed scheme resolves the issue of non-feasibility by processing large-scale data (big data) in a parallel fashion. Experiments have been demonstrated over real and synthetically generated large-scale datasets. Results evince that the proffered scheme is much more efficient and maintains the balance between the privacy preservation, data quality maintenance, and CPU time, when dealing with large voluminous big datasets compared to existing knowledge hiding techniques.

Performance Optimization Through Data Pipeline in Heterogenious Hadoop Cluster

Big data has received a momentum from both The scholarly group and organisation. The MapReduce version has risen into a noteworthy figuring mannequin on the aspect of large information research. Hadoop, that is an open supply utilization of the MapReduce mannequin, has been generally taken up by the network. Cloud expert businesses, for example, Amazon EC2 cloud have now upheld Hadoop client applications. no matter the whole lot, a key take a seem at is that the cloud educated co-ops do not a have asset provisioning tool to satisfy client occupations with due date prerequisites. As of now, it's miles completely the consumer duty to assess the require degree of property for his or her pastime running in an open cloud. This postulation correct-knownshows a Hadoop execution mannequin that exactly gauges the execution duration of exertions and in a similar manner arrangements the desired degree of property for a vocation to be finished indoors a due date. The proposed mannequin utilizes in the neighborhood Weighted Linear Regression (LWLR) mannequin to assess execution time of a vocation and Lagrange Multiplier device for asset provisioning to fulfill client art work with a given due date.

Spatial hotspot detection using polygon propagation

ABSTRACTSpatial scan statistics is one of the most important models in order to detect high activity or hotspots in real world applications such as epidemiology, public health, astronomy and criminology applications on geographic data. Traditional scan statistic uses regular shapes like circles to detect areas of high activity; the same model was extended to eclipses to improve the model. More recent works identify irregular shaped hotspots for data with geographical boundaries, where information about population within the geographical boundaries is available. With the introduction of better mapping technology, mapping individual cases to latitude and longitude became easier compared to aggregated data for which the previous models were developed. We propose an approach of spatial hotspot detection for point data set with no geographical boundary information. Our algorithm detects hotspots as a polygon made up of a set of triangles that are computed by a Polygon Propagation algorithm. The time complexity of the algorithm is non-linear to the number of observations, which does not scale well for larger datasets. To improve the model, we also introduce a MapReduce version of our algorithm to identify hotspots for larger datasets.

MR-I MaxMin-scalable two-phase border based knowledge hiding technique using MapReduce

Border based Knowledge hiding techniques (BB-KHT) are widely adopted form of privacy preservation techniques of data mining. These approaches are used to hide sensitive knowledge (confidential information) present in a dataset before sharing or analyzing it. BB-KHT primarily rely on border theory and maximum criterion method for preserving privacy and perpetuating good data quality of sanitized dataset but costs high computational complexity. Further, due to sequential nature, these approaches are particularly felicitous for small datasets and become infeasible while dealing with large scale datasets. Therefore, to subjugate the identified challenges of infeasibility and high computational complexity, a scalable two-phase improved MaxMin BB-KHT using MapReduce framework (MR-I MaxMin) is proposed. The proposed scheme requires only two database scans throughout the hiding process and hence, is computationally inexpensive. Moreover, the scheme also commits to preserve good data quality of sanitized dataset. The MapReduce version of proposed approach helps in achieving the feasibility by processing large voluminous data in a parallel fashion. Quantitative experiments and evaluations have been performed over a number of real and synthetically generated large-scale datasets. It is shown that the proposed MR-I MaxMin technique outperforms the similar existing approaches and vanquishes the identified challenges along with much-needed privacy preservation.

Scalable two-phase co-occurring sensitive pattern hiding using MapReduce

BackgroundExpansion of Internet and its use for on-line activities such as E-Commerce and social networking are producing large volumes of transactional data. This huge data volume resulted from these activities facilitates the analysis and understanding of global trends and interesting patterns used for several decisive purposes. Analytics involved in these processes expose sensitive information present in these datasets, which is a serious privacy threat. To overcome this challenge, few sequential heuristics have been used in past where volumes of data were comparatively accommodating to these sequential heuristics; the current situation is not that much in-line and often results in high execution time. This new challenge of scalability paves a way for experimenting with Big Data approaches (e.g., MapReduce Framework). We have agglomerated the MapReduce framework with adopted heuristics to overcome this challenge of scalability along with much-needed privacy preservation and yields efficient analytic results within bounded execution times.MethodsMapReduce is a parallel programming framework [16] which provides us the opportunity to leverage largely distributed resources to deal with the Big Data analytics. MapReduce allows the resource of a largely distributed system to be utilized in a parallel fashion. The simplicity and high fault-tolerance are the key features which make MapReduce a promising framework. Therefore, we have proposed a two-phase MapReduce version of these adopted heuristics. MapReduce framework divides the whole data into ‘n’ number of data chunks D = {d 1 d ∪ 2 ∪ d 3 ..... ∪ d n } and distributes them over ‘n’ computing nodes to achieve the parallelization. The first phase of MapReduce job runs on each data chunk in order to generate intermediate results, which are further sorted and merged in the second phase to generate final sanitized dataset.ResultsWe conducted three set of experiments, each with five different scenarios corresponding to the different cluster sizes i.e., n = 1,2,3,4,5 where ‘n’ is a number of computing nodes. We compared the approaches with respect to real as well as synthetically generated large datasets. For varying data sizes and varying number of computing nodes, it has been observed that sanitization time required by the MapReduce-based algorithm for same size dataset is much less than the sequential traditional approach. Further, the scalability can be improved by using more number of computing nodes. Lastly, another set of experiments explores the change in sanitization time with varying sizes of the sensitive content present in a dataset. We evaluated the effectiveness of proposed approach in different scenarios, with varying cluster size from 1 to 5 nodes. It has been observed that still the execution time of our approach is much less than traditional schemes. Further, no hiding failure, artifactual patterns have been observed during the experiments as well as in terms of misses cost also the MapReduce version performance is same as of traditional approaches.ConclusionTraditional approaches for data hiding primarily MaxFIA and SWA were lacking with due inability to tackle large voluminous data. To subjugate the new challenge of scalability, we have implemented these basic heuristics with Big Data approach i.e., MapReduce framework. Quantitative evaluations have shown that the fusion of MapReduce framework with these adopted heuristics fulfills its obligatory responsibility of being scalable and many-fold faster for yielding efficient analytic results.

A Framework for Exploring Algorithms for Big Data Mining

Objectives: To proposed and implement a framework that facilitates exploration of algorithms for big data mining. Methods/Analysis: To achieve objectives, a framework is built in order to realize algorithms for big data mining and even provide Mining as a Service in cloud. As the existing data mining techniques can not work for MapReduce programming in distributed environment, we proposed new and equivalent method for k-Anonymity that can leverage the parallel processing power. Single host Hadoop is used to demonstrate the proof of concept of the proposed framework. Findings: The framework has ability to mind big data. It has the centralized service that can be used to mine data of different users. However, as of now, the framework is realized with only one algorithm that is MapReduce version of k-Anonymity which is meant for privacy preserving data mining. The application is proved to be scalable in distributed environment. The framework has provision for supporting cloud users to outsource their data for mining big data with different algorithms of their choice. The results revealed that the proposed framework can provide mining services to cloud users and help them to save money by reusing the service instead of reinventing the wheel. Novelty/Improvement: In the proposed work a mining service for cloud is proposed which is a novel idea that has not been implemented so far. It can save money and time to enterprises in the real world.

English

Relentless Itemset Exhuming (RIE) is a archetypal data Exhuming topic with many authentic world applications such as market basket analysis. In authentic world the dataset size grows, researchers have prophesied Map Reduce version of RIE contrivance to meet the immensely colossal data challenge. Subsisting relentless itemset contrivance cannot distribute data equipollent among all the nodes and MR Apriori contrivance paper utilize manifold map/reduce procedures and engender an inordinate extent of key-value pairs with value. In this paper present a novel collateral, distributed contrivance which addresses these quandaries. We prophesis an ameliorated collateral contrivance and discuss its applications in this paper. In particular, we introduce a minute files processing strategy for massive minute files datasets to compensate defects of low read/indite speed and low processing efficiency in Hadoop. Moreover,we utilize MapReduce to implement the collateralization of FP-Magnification contrivance, thereby amending the overall performance of relentless itemsets exhuming. In the experiments we withal show that the prophesied contrivance returns a good performance compare with subsisting contrivance. FP-Magnification contrivance and discuss its applications in this paper. In

Iterative sampling based frequent itemset mining for big data

Frequent pattern mining attracts extensive research interests over the past two decades: including mining frequent item sets from transactions, extracting frequent sequences from bio-arrays and detecting common subgraph from molecular structures. In the era of big data, the explosive data volume brings new challenges to frequent pattern mining: (1) Space complexity: both input data, intermediate results and the outputted patterns could be too large to fit into memory which prevents many algorithms from executing; (2) Time complexity: many existing approaches rely on exhaustive search or complicated data structures to mine frequent patterns which prove to be inapplicable for big data. To deal with these two challenges. we propose ISbFIM, an Iterative Sampling based Frequent Itemset Mining method. Rather than process the entire data set at once, ISbFIM samples computationally-manageable subsets and extracts frequent itemsets from these subsets. By repeating this process for a sufficient number of times, we can guarantee both theoretically and empirically that the frequent itemsets can be enumerated without running into a combinatorial explosion. ISbFIM can be easily parallelized and applied to mine item sets, sequences or structures. We implement a Map-Reduce version of ISbFIM to demonstrate its scalability on big data.

Scaling up MapReduce-based Big Data Processing on Multi-GPU systems

MapReduce is a popular data-parallel processing model encompassed with recent advances in computing technology and has been widely exploited for large-scale data analysis. The high demand on MapReduce has stimulated the investigation of MapReduce implementations with different architectural models and computing paradigms, such as multi-core clusters, Clouds, Cubieboards and GPUs. Particularly, current GPU-based MapReduce approaches mainly focus on single-GPU algorithms and cannot handle large data sets, due to the limited GPU memory capacity. Based on the previous multi-GPU MapReduce version MGMR, this paper proposes an upgrade version MGMR++ to eliminate GPU memory limitation and a pipelined version, PMGMR, to handle the Big Data challenge through both CPU memory and hard disks. MGMR++ is extended from MGMR with flexible C++ templates and CPU memory utilization, while PMGMR fine-tuned the performance through the latest GPU features such as streams and Hyper-Q as well as hard disk utilization. Compared to MGMR (Jiang et al., Cluster Computing 2013), the proposed schemes achieve about 2.5-fold performance improvement, increase system scalability, and allow programmers to write straightforward MapReduce code for Big Data.

An Algorithm for Mining Frequent Itemsets from Library Big Data

Frequent itemset mining plays an important part in college library data analysis. Because there are a lot of redundant data in library database, the mining process may generate intra-property frequent itemsets, and this hinders its efficiency significantly. To address this issue, we propose an improved FP-Growth algorithm we call RFP-Growth to avoid generating intra-property frequent itemsets, and to further boost its efficiency, implement its MapReduce version with additional prune strategy. The proposed algorithm was tested using both synthetic and real world library data, and the experimental results showed that the proposed algorithm outperformed existing algorithms.

Efficient Sequence Alignment Using MapReduce on the Cloud

Abstract Over the past few years, advances in the field of molecular biology and genomic technologies have led to an explosive growth of digital biological information. The analysis of this large amount of data is commonly based on the extensive and repeated use of conceptually parallel algorithms, most notably in the context of sequence alignment. Cloud computing provides scientists with a completely new model of utilizing the computing infrastructure. Cloud computing model is excellent in dealing with such bioinformatics applications, which require both management of huge amounts of data and heavy computations. The study aims at transforming a recently developed bioinformatics sequence alignment tool, named BFAST, to the cloud environment. The MapReduce version of the BFAST tool will be used to demonstrate the effectiveness of the MapReduce framework and the cloud-computing model in handling the intensive computations and management of the huge bioinformatics data. A number of existing tools and technologies are utilized in this study to achieve an efficient transformation of the BFAST tool into the cloud environment. The implementation is mainly based on two core components; BFAST and MapReduce. BFAST is a software package for aligning next generation genomic reads against a target genome with a very high accuracy and reasonable speed. MapReduce general-purpose parallelization technology [in its open source implementation, Hadoop] appears to be particularly well adapted to the intensive computations and huge data storage tasks involved in the BFAST sequence alignment tool. The MapReduce version of the BFAST tool is expected to offer better results than the original one in terms of maintaining good computational efficiency, accuracy, scalability, deployment and management efforts. The study demonstrates how a general-purpose parallelization technology, i.e. MapReduce running on the cloud, can be tailored to tackle the class of bioinformatics problems with good performance and scalability, and, more importantly, how this technology could be the basis of a computational parallel platform for several problems in the context of bioinformatics. Although the effort of transforming existing bioinformatics algorithms from local compute infrastructure is not trivial, the speed and flexibility of cloud computing environments provide a substantial boost with manageable cost.

MapReduce in MPI for Large-scale graph algorithms

We describe a parallel library written with message-passing (MPI) calls that allows algorithms to be expressed in the MapReduce paradigm. This means the calling program does not need to include explicit parallel code, but instead provides “map” and “reduce” functions that operate independently on elements of a data set distributed across processors. The library performs needed data movement between processors. We describe how typical MapReduce functionality can be implemented in an MPI context, and also in an out-of-core manner for data sets that do not fit within the aggregate memory of a parallel machine. Our motivation for creating this library was to enable graph algorithms to be written as MapReduce operations, allowing processing of terabyte-scale data sets on traditional MPI-based clusters. We outline MapReduce versions of several such algorithms: vertex ranking via PageRank, triangle finding, connected component identification, Luby’s algorithm for maximally independent sets, and single-source shortest-path calculation. To test the algorithms on arbitrarily large artificial graphs we generate randomized R-MAT matrices in parallel; a MapReduce version of this operation is also described. Performance and scalability results for the various algorithms are presented for varying size graphs on a distributed-memory cluster. For some cases, we compare the results with non-MapReduce algorithms, different machines, and different MapReduce software, namely Hadoop. Our open-source library is written in C++, is callable from C++, C, Fortran, or scripting languages such as Python, and can run on any parallel platform that supports MPI.

CPRS: A cloud-based program recommendation system for digital TV platforms

Traditional electronic program guides (EPGs) cannot be used to find popular TV programs. A personalized digital video broadcasting-terrestrial (DVB-T) digital TV program recommendation system is ideal for providing TV program suggestions based on statistics results obtained from analyzing large-scale data. The frequency and duration of the programs that users have watched are collected and weighted by data mining techniques. A large dataset produces results that best represent a viewer’s preferences of TV programs in a specific area. To process such a massive amount of viewer preference data, the bottleneck of scalability and computing power must be removed. In this paper, an architecture for a TV program recommendation system based on cloud computing and a map-reduce framework, the map-reduce version of k -means and the k -nearest neighbor (kNN) algorithm, is introduced and applied. The proposed architecture provides a scalable and powerful backend to support the demand of large-scale data processing for a program recommendation system.