MapReduce Applications Research Articles

TEE-MR: Developer-friendly data oblivious programming for trusted execution environments

Trusted execution environments (TEEs) enable efficient protection of integrity and confidentiality for applications running on untrusted platforms. They have been deployed in cloud servers to attract users who have concerns on exporting data and computation. However, recent studies show that TEEs’ side channels, including memory, cache, and micro-architectural features, are still vulnerable to adversarial exploitation. As many such attacks utilize program access patterns to infer secret information, data oblivious programs have been considered a practical defensive solution. However, they are often difficult to develop and optimize via either manual or automated approaches. We present the oblivious TEE with MapReduce (TEE-MR) approach that uses application frameworks, an approach between fully manual and fully automated, to hide the details of access-pattern protection to significantly minimize developers’ efforts. We have implemented the approach with the MapReduce application framework for data-intensive applications. It can regulate application dataflows and hide application-agnostic access-pattern protection measures from developers. Compared to manual composition approaches, it demands much less effort for developers to identify access patterns and to write code. Our approach is also easy to implement, less complicated than fully automated approaches, for which we have not seen a working prototype yet. Our experimental results show that TEE-MR-based applications have good performance, comparable to those carefully developed with time-consuming manual composition approaches.

Automatic Debugging of Design Faults in MapReduce Applications

Automatic Debugging of Design Faults in MapReduce Applications

IDaPS — Improved data-locality aware data placement strategy based on Markov clustering to enhance MapReduce performance on Hadoop

The execution of Map-Reduce applications on the Hadoop cluster poses significant challenges due to the non-consideration of data locality, i.e., assigning tasks to compute nodes where input data sets are located. Due to such non-consideration, high data transfer overheads are caused. Further, it increases latency, which may arise if input data needs to be transferred across the network, thereby significantly increasing execution time. To address this issue, an Improved DAta Placement Strategy IDaPS based on the intra-dependency among the data is proposed. IDaPS re-organizes the default data layouts in HDFS to ensure higher degree of parallelism. The efficiency of IDaPS is demonstrated in Hadoop clusters (10 and 15 nodes) by executing Hadoop Benchmark performance tests viz. WordCount, Grep on Project-Gutenberg book dataset (50 GB) and Least Square Linear Regression (LSLR) on weather dataset (10.67 GB). The results were compared with state-of-the-art algorithms viz. Hadoop Default Data Placement (HDDP), Load-Balancer and literary work RENDA. The results demonstrate that IDaPS significantly reduces execution time by 28.2% and 38.4% in 10-node and 15-node clusters while executing WordCount, and 35% and 38.1% in 10-node and 15-node clusters for Grep. Similarly, for LSLR, it reduces execution time by 32.7%.

CREATING A HIGH-PERFORMANCE HADOOP CLUSTER CONSIST OF WEAK COMPUTERS WITHIN THE ORGANIZATION

Knowledge of practical methods of processing large amounts of data is akin to black magic. There are many tools and techniques for scalable data processing, in particular, caching (for example, using the mem cached program), replication, partitioning and, of course, Map Reduce/Hadoop. Hadoop is an open source framework that implements the Map Reduce distribution and reduction algorithm, which underlies Google's approach to organizing queries to distributed datasets that make up the Internet. This article is intended for programmers, architects and project managers involved in processing large amounts of data offline. It will describe how to get a copy of Hadoop, how to organize a cluster and how to write analysis programs. We will start by applying Hadoop in the default configuration to solving a few simple tasks, for example, analyzing changes in the frequency of occurrence of words in the corpus of documents; this will help to understand the basic ideas of Hadoop and MapReduce. Next, we will move on to the basic concepts of MapReduce applications developed using Hadoop, and along the way we will study the components of the framework, the application of Hadoop to a wide range of data analysis tasks and numerous examples of Hadoop in action. Keywords: big data, cluster, Hadoop.

New efficient fractal models for MapReduce in OpenMP parallel environment

Parallel data processing is one of the specific infrastructure applications categorized as a service provided by cloud computing. In cloud computing environments, data-intensive applications increasingly use the parallel processing paradigm known as MapReduce. MapReduce is based on a strategy called "divide and conquer," which uses ordinary computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used in the MapReduce application using proposed fractal network models. Two fractal network models are offered, and their work is compared with a well-known network model, the hypercube. The first fractal network model achieved an average speedup of 3.239 times while an efficiency ranged from 73-95%. In the second model of the network, the speedup got to 3.236 times while keeping an efficiency of 70-92%. Furthermore, the path-finding algorithm employed in the recommended fractal network models remarkably identified all paths and calculated the shortest and longest routes.

New efficient fractal models for MapReduce in OpenMP parallel environment

Parallel data processing is one of the specific infrastructure applications categorized as a service provided by cloud computing. In cloud computing environments, data-intensive applications increasingly use the parallel processing paradigm known as MapReduce. MapReduce is based on a strategy called "divide and conquer," which uses ordinary computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used in the MapReduce application using proposed fractal network models. Two fractal network models are offered, and their work is compared with a well-known network model, the hypercube. The first fractal network model achieved an average speedup of 3.239 times while an efficiency ranged from 73-95%. In the second model of the network, the speedup got to 3.236 times while keeping an efficiency of 70-92%. Furthermore, the path-finding algorithm employed in the recommended fractal network models remarkably identified all paths and calculated the shortest and longest routes.

An innovativefractal architecture model for implementing MapReduce in an open multiprocessing parallel environment

One of the infrastructure applications that cloud computing offers as a service is parallel data processing. MapReduce is a type of parallel processing used more and more by data-intensive applications in cloud computing environments. MapReduce is based on a strategy called "divide and conquer," which uses regular computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used with the proposed fractal network model in the MapReduce application. A well-known model, the cube, is used to compare the fractal network model and its work. Where experiments demonstrated that the fractal model is preferable to the cube model. The fractal model achieved an average speedup of 2.7 and an efficiency rate of 67.7%. In contrast, the cube model could only reach an average speedup of 2.5 and an efficiency rate of 60.4%.

Multi-Tenancy- and Redundancy-Aware In-Network Aggregation using Programmable Switches

Recent advances in programmable switches make it possible to aggregate data in partition-aggregation applications (e.g., MapReduce applications) at programmable switches. However, a well-designed aggregation scheme is indispensable for aggregating as much data as possible under the limited resources of programmable switches in an environment where multiple applications are running concurrently. In this article, we propose a multitenancy- and redundancy-aware in-network aggregation (MARINA) scheme that preferentially aggregates highly redundant data at a programmable switch and improves aggregation performance by constructing a multi-tenancyaware aggregation tree. Evaluation results demonstrate that MARINA can improve data aggregation performance by up to 81% compared with the conventional approach using statically partitioned resources and sequential aggregation.

Lightweight MapReduce Application Service Integrity Auditing on the Cloud

Due to the advantages of MapReduce in both cost-saving and effectiveness for large-scale computation-intensive applications, a growing number of tenants are considering migrating their applications to the MapReduce cloud. For financial interests, an untrusted cloud service provider (CSP) may violate the service level agreement (SLA) when providing service to tenants. Therefore, essential security mechanisms are needed to protect the integrity of MapReduce services. To address this issue, a novel service accountability scheme is proposed to audit compliance for MapReduce on the cloud in this paper. The proposed scheme inserts auditing points into tenant’s applications and constructs a tamper-proof list at run time. By verifying the auditing list, the MapReduce cloud can be responsible and accountable, which can detect whether CSP should be held responsible for its service. Finally, the proposed scheme is deployed in experimental business processes, and its effectiveness is evaluated.

Using software visualization to support the teaching of distributed programming

In this paper, we introduce MARVEL, a system designed to simplify the teaching of MapReduce, a popular distributed programming paradigm, through software visualization. At its core, it allows a teacher to describe and recreate a MapReduce application by interactively requesting, through a graphical interface, the execution of a sequence of MapReduce transformations that target an input dataset. Then, the execution of each operation is illustrated on the screen by playing an appropriate graphical animation stage, highlighting aspects related to its distributed nature. The sequence of all animation stages, played back one after the other in a sequential order, results in a visualization of the whole algorithm. The content of the resulting visualization is not simulated or fictitious, but reflects the real behavior of the requested operations, thanks to the adoption of an architecture based on a real instance of a distributed system running on Apache Spark. On the teacher’s side, it is expected that by using MARVEL he/she will spend less time preparing materials and will be able to design a more interactive lesson than with electronic slides or a whiteboard. To test the effectiveness of the proposed approach on the learner side, we also conducted a small scientific experiment with a class of volunteer students who formed a control group. The results are encouraging, showing that the use of software visualization guarantees students a learning experience at least equivalent to that of conventional approaches.

SwMR: A Framework for Accelerating MapReduce Applications on Sunway Taihulight

MapReduce has already been an indispensable framework for the programmers to develop big data applications at scale without concerning the underlying complex system details. However, such an important framework is missing on the Sunway many-core processor that powers the world-leading supercomputer Sunway Taihulight. This paper fills the gap by implementing an efficient MapReduce framework on Sunway many-core processor which takes full advantage of the architecture features such as local device memory and register communication. Specifically, we propose three different schemes that adopt different methods to partition the computation of map and reduce across the MPE and CPEs of Sunway processor. Especially in CPE-accelerated Dynamic Partitioning Scheme (CDPS), the processing role of the CPEs within each CPE pair can be transformed dynamically between map and reduce, which is effective to improve the load balance during runtime. In addition, we adopt the local device memory as well as register communication on each CPE to improve the efficiency of data access and communication. The experiment results demonstrate our MapReduce framework based on CDPS achieves 36.9× performance speedup on average across representative benchmarks.

Research on the Method and Application of MapReduce in Mobile Track Big Data Mining

Background: Mass movement trajectory data with real scenarios has been evolved with big data mining to solve the data redundancy problem. Methods: This paper proposes a parallel path based on the Map Reduce compression method, using two kinds of piecewise point mutual crisscross, the classified method of trajectory, and then segment trajectory distribution to multiple nodes to parallelize the compression. Results: Finally, the results based on both compression methods have been simulated for the different real-time data by merging both techniques. Conclusion: The performance test results show that the parallel trajectory compression method proposed in this paper can greatly improve the compression efficiency and completely eliminate the error caused by the failure of the correlation between the segments.

Parameter Settings Optimization in Map Reduce Big Data processing using the MOPSO Algorithm

Big data is a commodity that is highly valued in the entire globe. It is not just regarded as data but in the world of experts, we can derive intelligence from it. Because of its characteristics which are Variety, Value, Volume, Velocity, and the growing need of how it can be handled, Organizations are facing difficulties in ensuring optimal as well as affordable processing and storage of large datasets. One of the already existing models used for rapid processing together with storage in big data is known as Hadoop MapReduce. MapReduce is used for large-scale data processing in a parallel and distributed computing environment, while Hadoop is used for running applications and storing data in clusters of commodity hardware Furthermore, the Hadoop MapReduce framework needs to tune more than 190 configuration parameters which are mostly done manually. Due to complex interactions and large spaces between parameters, manual tuning is not effective. Even worse, these parameters must be tuned every time Hadoop MapReduce applications are run. The main goal of this research is to create an algorithm that will improve efficiency by automatically optimizing parameter settings when MapReduce jobs are running. The algorithm employs the Multi-Objective Particle Swarm Optimization (MOPSO) technique, which uses two objective functions to look for a Pareto optimal solution while optimizing the parameters. The results of the experiments have shown that the algorithm has remarkably improved MapReduce job performance in comparison to the use of default settings.

BigDataSDNSim: A simulator for analyzing big data applications in software‐defined cloud data centers

AbstractThe integration and crosscoordination of big data processing and software‐defined networking (SDN) are vital for improving the performance of big data applications. Various approaches for combining big data and SDN have been investigated by both industry and academia. However, empirical evaluations of solutions that combine big data processing and SDN are extremely costly and complicated. To address the problem of effective evaluation of solutions that combine big data processing with SDN, we present a new, self‐contained simulation tool named BigDataSDNSim that enables the modeling and simulation of the big data management system YARN, its related programming models MapReduce, and SDN‐enabled networks in a cloud computing environment. BigDataSDNSim supports cost‐effective and easy to conduct experimentation in a controllable, repeatable, and configurable manner. The article illustrates the simulation accuracy and correctness of BigDataSDNSim by comparing the behavior and results of a real environment that combines big data processing and SDN with an equivalent simulated environment. Finally, the article presents two uses cases of BigDataSDNSim, which exhibit its practicality and features, illustrate the impact of data replication mechanisms of MapReduce in Hadoop YARN, and show the superiority of SDN over traditional networks to improve the performance of MapReduce applications.

Methodology for Modified Whale Optimization Algorithm for Solving Appliances Scheduling Problem

Whale Optimization Algorithm (WOA) is considered as one of the newest metaheuristic algorithms to be used for solving a type of NP-hard problems. WOA is known of having slow convergence and at the same time, the computation of the algorithm will also be increased exponentially with multiple objectives and huge request from n users. The current constraints surely limit for solving and optimizing the quality of Demand Side Management (DSM) case, such as the energy consumption of indoor comfort index parameters which consist of thermal comfort, air quality, humidity and vision comfort. To address these issues, this proposed work will firstly justify and validate the constraints related to the appliances scheduling problem, and later proposes a new model of the Cluster based Multi-Objective WOA with multiple restart strategy. In order to achieve the objectives, different initialization strategy and cluster-based approaches will be used for tuning the main parameter of WOA under different MapReduce application which helps to control exploration and exploitation, and the proposed model will be tested on a set of well-known test functions and finally, will be applied on a real case project i.e. appliances scheduling problem. It is anticipating that the approach can expedite the convergence of meta-heuristic technique with quality solution.

Scaling of Hadoop Cluster for Cost-Effective Processing of MapReduce Applications

Scaling of Hadoop Cluster for Cost-Effective Processing of MapReduce Applications

Transparent many‐core partitioning for high‐performance big data I/O

SummaryAs the number of cores equipped in a single computing node is rapidly increasing, utilizing many cores for contemporary applications in an efficient manner is a challenging issue. We need to consider both parallelization and locality to fully exploit many cores for multifarious operations of emerging applications. In particular, big data applications perform computation and I/O intensive operations alternately. For instance, Apache Hadoop MapReduce assumes local persistent storage for each computing node. Thus, unlike traditional parallel programming models, the MapReduce framework performs not only networking but also storage I/O. In this study, we aim to improve the locality of network and storage I/O operations on many‐core systems by partitioning cores for I/O system calls and event handlers. In order to implement fine‐grained many‐core partitioning, we decouple the system call context from the user‐level process by suggesting message‐based system calls. The suggested design provides user‐level transparency and does not require any kernel‐level modifications. In addition, we propose a scheme that dynamically decides the core affinity of system calls and event handlers by considering locality, run‐time loads, and hardware architectures. The experimental results show that the proposed many‐core partitioning can improve the locality of network and storage I/O operations in an integrated manner for MapReduce applications.

A comparison of forecasting models for the resource usage of MapReduce applications

In this paper, we construct forecasting models (multivariate long short-term memory recurrent neural networks and multiple linear regression) for the resource usage prediction of four MapReduce applications and applications executed within the Spark framework. We have evaluated the impact of a sample size to prediction accuracy. Also, we propose a phase modelling approach for read/write-intensive applications. Our results show that models based on long short-term memory recurrent neural networks exhibit a higher accuracy than multiple linear regression models and the intensive characteristics of a resource are closely related to the prediction accuracy of forecasting models. We investigated the hyperparameter tuning of such models and showed that a randomly initialised, shallow, well-tuned network may outperform deeper models that use stacked autoencoder initialisation. Furthermore, multivariate long short-term memory recurrent neural network models are more sensitive to sample size than multiple linear regression models. We show that an LSTM model trained in a specific machine may be used to predict the resource usage in another machine.

QoS-Aware Data Placement for MapReduce Applications in Geo-Distributed Data Centers

With growing data volumes and the scaling of data center clusters, communication resources often become a bottleneck in service provisioning for many MapReduce applications (e.g., training machine learning models). Therefore, data placements that bring data blocks closer to data consumers (e.g., MapReduce applications) are seen as a promising solution. In this article, we propose an efficient data-placement technique that considers network traffic reduction as well as QoS guarantees for the data blocks to optimize the communication resources. We first formulate the joint optimization of the data-placement problem, propose a generic model for minimizing communication costs, and show that the joint data-placement problem is NP-hard. To solve this problem, we propose a heuristic algorithm considering traffic flows in the network topology of data centers by first seeking optimal QoS-aware data placement based on golden division on a Zipflike replica distribution, then transforming the joint data-placement problem into a block-dependence tree (BDT) construction problem, and finally reducing the BDT construction to a graph-partitioning problem. The experimental results demonstrate that our data-placement approach could effectively improve the performance of MapReduce jobs with lower communication costs and less job execution time for big-data processing.

A novel approach for improving data locality of MapReduce applications in cloud environment through intelligent data placement

In this world of big data, hosting storage and analytics as cloud service is extremely relevant. In multi-user environments, there are chances for load imbalance during data placement. MapReduce like frameworks move computation towards data. However, because of load imbalance, some nodes cannot start computation on the node on which data is stored and may be compelled to start computation on some other nodes. This results in deteriorating data locality. In this case, data have to be copied to the computing node. This data transfer increases the job completion time. This paper proposes a data placement policy for clouds in which the data and virtual machines are collocated in the same set of physical servers. The physical servers in the cloud are grouped into partitions created using the minimum spanning tree. Experimental results show that this proposal improves node utilisation and reduces execution time over default placement in the cloud environment.