MapReduce System Research Articles

Design of an Iterative Method for MapReduce Scheduling Using Deep Reinforcement Learning and Anomaly Detection

Due to high complexities within distributed computing environments, there's a critical need for advanced scheduling frameworks that are capable of optimizing MapReduce systems. Current approaches have static policies that limit their capability to adapt to changing system dynamics and workload variations for different cloud scenarios. To overcome these issues, this study introduces a robust MapReduce framework empowered by intelligent scheduling algorithms, tailored to enhance system efficiency and resilience levels. The framework introduces three novel scheduling models: Deep Reinforcement Learning for Dynamic Job Scheduling (DRDJS), Anomaly Detection-driven Adaptive Scheduling (ADAS), and Cluster-based Job Categorization and Scheduling (CJCS). DRDJS utilizes deep reinforcement learning to dynamically generate optimal scheduling policies based on multiple metrics that include historical job data, system metrics, and workload characteristics. This adaptive approach leads to significant reductions in job completion times, outperforming static scheduling methods by up to 30%. Next, ADAS leverages anomaly detection to prioritize critical tasks and efficiently allocate resources in response to anomalies, resulting in a notable improve scheduling speed by up to 40%. Finally, CJCS employs clustering algorithms to categorize jobs based on their resource requirements and execution profiles, enabling more accurate resource allocation and reducing average job completion times by up to 25%. By integrating these methods into a unified scheduling framework, the proposed solution addresses the variability in job characteristics and system performance, thereby enhancing the overall throughput and stability of MapReduce operations.

SARS: Towards minimizing average Coflow Completion Time in MapReduce systems

Reducing the average Coflow Completion Time (CCT) plays a vital role to improve the responsiveness and throughput of distributed computing systems. However, most of the previous works on minimizing the average CCT scheduled all coflows following the Shortest Remaining Time First (SRTF) principle. In this paper, we argue that SRTF is not a necessary (even not a good) principle to follow in order to minimize the average CCT. Instead, we propose a Simulated Annealing based Reducer placement and coflow Scheduling approach named SARS to minimize the average CCT in MapReduce systems. In SARS, two powerful modules, i.e., the Neighboring State Searching (NSS) module and the System Energy Estimation (SEE) module, are designed to search a better coflow scheduling order and estimate the minimum average CCT given the scheduling order, respectively. Based on the interaction between these two modules, a near-optimal reducer placement and coflow scheduling scheme will be derived. Both testbed experiments and extensive simulations show that SARS can reduce the average CCT by up to 64.32% compared with the state-of-the-art technique.

In-Storage Computing for Hadoop MapReduce Framework: Challenges and Possibilities

Solid State Drives (SSDs) were initially developed as faster storage devices intended to replace conventional magnetic Hard Disk Drives (HDDs). However, high computational capabilities enable SSDs to be computing nodes, not just faster storage devices. Such capability is generally called ”In-Storage Computing (ISC)”. Today’s Hadoop MapReduce framework has become a de facto standard for big data processing. This paper explores In-Storage Computing challenges and opportunities for the Hadoop MapReduce framework. For this, we integrate a Hadoop MapReduce system with ISC SSD devices that implement the Hadoop Mapper inside real SSD firmware. This offloads Map tasks from the host MapReduce system to the ISC SSDs. We additionally optimize the host Hadoop system to make the best use of our proposed ISC Hadoop system. Experimental results demonstrate our ISC Hadoop MapReduce system achieves a remarkable performance gain (2.3 faster) as well as significant energy savings (11.5 lower) compared to a typical Hadoop MapReduce system. Further, the experiment suggests such ISC augmented systems can provide a very promising computing model in terms of a system scalability.

Enhancing Data Engineering Frameworks for Scalable Real-Time Marketing Solutions

This report discusses improvements to the data engineering frameworks for cost-efficient real-time marketing environment, with the applications in mobile advertising and smart city analytics. The proposed framework combines the processing of time-series data with map-reduce (M-R) systems in order to compensate for inefficiencies in the execution of temporal queries and the processing of real-time data streams. It enables effective behavioral targeting in mobile marketing and helps integrate various domains’ real-time data for smart city utilization. Specific examples that illustrate further refinements in the algorithms are showcased through case study and experimental evidence where gains in throughput, data reliability, and system effectiveness are quantified in real-time Marketing and Smart City applications.

Optimized hadoop map reduce system for strong analytics of cloud big product data on amazon web service

Because of the rapid increase of data in the cloud of Amazon Web Service (AWS), the traditional methods for analyzing this data are not good and inappropriate, so unconventional methods of analysis have been proposed by many data scientists such as concurrent/ parallel techniques to meeting the requirements of performance and scalability entailed in such big data analyses. In this paper we are used Hadoop Map Reduce system that contains Hadoop Distributed File System (HDFS) and Hadoop cluster. We optimized it by combining it with five efficient Data Mining (DM) algorithms such as Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), Correlative Naïve Bayes classifier (CNB), and Fuzzy CNB (FCNB) for strong analytics of cloud big data. The proposed system applied on product review data that taken form the cloud of AWS. The Evaluation of Hadoop Map Reduce done with important benchmarks as Mean Absolute Percentage Error (MPAE), Root Mean Square Error (RMSE), and runtime for word count, sort, inverted index. Also, the evaluation of DM models with Hadoop Map Reduce system done by using accuracy, sensitivity, specificity, memory, and running time. Experiments have shown that FCNB is effective in addressing the problem of big data.

Top-K Query for Large Dataset of Restaurant Review Based-on Hadoop MapReduce Framework

In order to develop post-COVID-19 culinary tourism, high-quality facilities and services are essential. Information technology can contribute through a top-k query-based decision-making system. This study implements top-k queries on a distributed Hadoop MapReduce system to evaluate its capability in managing data and selecting culinary tourism potential. Research findings indicate that for the European restaurant data from TripAdvisor with 5 dimensions, both single-node and multi-node (3 nodes) executions exhibit comparable execution times across various data quantities. Conversely, for the European restaurant data from TripAdvisor with 14 dimensions, the use of multi-node (3 nodes) tends to result in longer execution times compared to the single-node approach for larger data quantities. Furthermore, the utilization of multi-node (3 nodes) proves to be more efficient in processing synthetic data with 5 dimensions as the data quantity increases, demonstrating a significant difference in execution times compared to the single-node approach. The study also reveals that across different dimensions, the multi-node (3 nodes) approach generally outperforms the single-node approach in terms of speed. Regarding node variations, processing 20 million data points with 5 dimensions using 6 nodes yields the optimal method with the shortest execution time. By leveraging information technology and a top-k query-based decision-making system, the development of culinary tourism potential can be conducted more efficiently and effectively. The performance of MapReduce in processing culinary tourism potential data can be optimized by employing multi-node execution for large datasets and single-node execution for relatively smaller datasets.

A Complexity Aware Scheduler With Dynamic Slot Allocation for Cloud Video Transcoding

For a multimedia cloud computing platform, it needs to perform video transcoding to provide bandwidth-compatible bit-streams for users. In configuring a MapReduce system, it allocates slots to workers with the assumption of homogeneous worker power and performs task scheduling by assuming equal task time complexity. However, the computing power of a practical cluster of workers and the time complexity of tasks is time-varying in any case. The task-scheduling algorithm has to well manipulate these heterogeneous cloud resources and task complexity. We first find a good partition size for a video transcoding job for efficient processing. We proposed a Complexity-Aware Scheduling (CAS) algorithm that reorders task assignment priorities according to task complexity to maintain load-balancing operations. By utilizing a Neural Network model to refine the task complexity estimation for the CAS(CASNN), the scheduling and transcoding speedup performances can further be improved. Based on the CASNN, we proposed a Dynamically Adjusting Slot number allocation (DAS) method, DASCASNN, to adjust the slots according to resource utilization status to improve the processing performance. Experimental results show the proposed DASCASNN can help reduce 30% of the transcoding time on average as compared to available schedulers and increase the resource utilization rates from 82.7% to 98%.

Parallel news clustering and topic modeling approaches

At the current age there is an urgent need in developing massively scalable and efficient tools to Big Data processing. Even the smallest companies nowadays inevitably require more and more resources for data processing routines that could enhance decision making and reliably predict and simulate different scenarios. In the current paper we present our combined work on different massively scalable approaches for the task of clustering and topic modeling of the dataset, collected by crawling Kazakhstan news websites. In particular, we propose Apache Spark parallel solutions to news clustering and topic modeling problems and, additionally, we describe results of implementing document clustering using developed partitioned global address space Mapreduce system. In our work we describe our experience in solving these problems and investigate the efficiency and scalability of the proposed solutions.

Efficient verification of parallel matrix multiplication in public cloud: the MapReduce case

With the advent of cloud-based parallel processing techniques, services such as MapReduce have been considered by many businesses and researchers for different applications of big data computation including matrix multiplication, which has drawn much attention in recent years. However, securing the computation result integrity in such systems is an important challenge, since public clouds can be vulnerable against the misbehavior of their owners (especially for economic purposes) and external attackers. In this paper, we propose an efficient approach using Merkle tree structure to verify the computation results of matrix multiplication in MapReduce systems while enduring an acceptable overhead, which makes it suitable in terms of scalability. Using the Merkle tree structure, we record fine-grained computation results in the tree nodes to make strong commitments for workers; they submit a commitment value to the verifier which is then used to challenge their computation results’ integrity using elected input data as verification samples. Evaluation outcomes show significant improvements comparing with the state-of-the-art technique; in case of 300*300 matrices, 73% reduction in generated proof size, 61% reduction in the proof construction time, and 95% reduction in the verification time.

Distributed Interactive Visualization Using GPU-Optimized Spark.

With the advent of advances in imaging and computing technologies, large-scale data acquisition and processing have become commonplace in many science and engineering disciplines. Conventional workflows for large-scale data processing usually rely on in-house or commercial software that are designed for domain-specific computing tasks. Recent advances in MapReduce, which was originally developed for batch processing textual data via a simplified programming model of the map and reduce functions, have expanded its applications to more general tasks in big-data processing, such as scientific computing, and biomedical image processing. However, as shown in previous work, volume rendering and visualization using MapReduce is still considered challenging and impractical owing to the disk-based, batch-processing nature of its computing model. In this article, contrary to this common belief, we show that the MapReduce computing model can be effectively used for interactive visualization. Our proposed system is a novel extension of Spark, one of the most popular open-source MapReduce frameworks, which offers GPU-accelerated MapReduce computing. To minimize CPU-GPU communication and overcome slow, disk-based shuffle performance, the proposed system supports GPU in-memory caching and MPI-based direct communication between compute nodes. To allow for GPU-accelerated in-situ visualization using raster graphics in Spark, we leveraged the CUDA-OpenGL interoperability, resulting in faster processing speeds by several orders of magnitude compared to conventional MapReduce systems. We demonstrate the performance of our system via several volume processing and visualization tasks, such as direct volume rendering, iso-surface extraction, and numerical simulations with in-situ visualization.

Parallel computation of probabilistic skyline queries using MapReduce

In recent years, numerous applications have been continuously generating large amounts of uncertain data. The advanced analysis queries such as skyline operators are essential topics to extract interesting objects from the vast uncertain dataset. Recently, the MapReduce system has been widely used in the area of big data analysis. Although the probabilistic skyline query is not decomposable, it does not make sense to implement the probabilistic skyline query in the MapReduce framework. This paper proposes an effective parallel method called parallel computation of probabilistic skyline query (PCPS) that can measure the probabilistic skyline set in one MapReduce computation pass. The proposed method takes into account the critical sections and detects data with a high probability of existence through a proposed smart sampling algorithm. PCPS implements a new approach to the fair allocation of input data. The experimental results indicate that our proposed approach can not only reduce the processing time of the probabilistic skyline queries, but also achieve fair precision with varying dimensionality degrees.

An Overview of Platforms for Big Earth Observation Data Management and Analysis

In recent years, Earth observation (EO) satellites have generated big amounts of geospatial data that are freely available for society and researchers. This scenario brings challenges for traditional spatial data infrastructures (SDI) to properly store, process, disseminate and analyze these big data sets. To meet these demands, novel technologies have been proposed and developed, based on cloud computing and distributed systems, such as array database systems, MapReduce systems and web services to access and process big Earth observation data. Currently, these technologies have been integrated into cutting edge platforms in order to support a new generation of SDI for big Earth observation data. This paper presents an overview of seven platforms for big Earth observation data management and analysis—Google Earth Engine (GEE), Sentinel Hub, Open Data Cube (ODC), System for Earth Observation Data Access, Processing and Analysis for Land Monitoring (SEPAL), openEO, JEODPP, and pipsCloud. We also provide a comparison of these platforms according to criteria that represent capabilities of the EO community interest.

Rough Sets Base Associative Classification Rules Extraction from Big Data

Big Data is a current burning challenge for the data analytics research community. Many conventional data analytics techniques have been extended to the MapReduce framework to process Big Data. But in our literature review, we find that for the MapReduce system there is an absolute lack of rough setbased technique. To facilitate this and recognize the importance of the rule-based classification techniques, we suggest a roughset associative classification rules extraction process for the MapReduce framework. The implementation and evaluation of the Big Data Standard data set demonstrated the efficiency of our suggested approach.

A New Framework for Evaluating Straggler Detection Mechanisms in MapReduce

Big Data systems (e.g., Google MapReduce, Apache Hadoop, Apache Spark) rely increasingly on speculative execution to mask slow tasks, also known as stragglers, because a job’s execution time is dominated by the slowest task instance. Big Data systems typically identify stragglers and speculatively run copies of those tasks with the expectation that a copy may complete faster to shorten job execution times. There is a rich body of recent results on straggler mitigation in MapReduce. However, the majority of these do not consider the problem of accurately detecting stragglers. Instead, they adopt a particular straggler detection approach and then study its effectiveness in terms of performance, e.g., reduction in job completion time or higher efficiency, e.g., high resource utilization. In this article, we consider a complete framework for straggler detection and mitigation. We start with a set of metrics that can be used to characterize and detect stragglers including Precision, Recall, Detection Latency, Undetected Time, and Fake Positive. We then develop an architectural model by which these metrics can be linked to measures of performance including execution time and system energy overheads. We further conduct a series of experiments to demonstrate which metrics and approaches are more effective in detecting stragglers and are also predictive of effectiveness in terms of performance and energy efficiencies. For example, our results indicate that the default Hadoop straggler detector could be made more effective. In a certain case, Precision is low and only 55% of those detected are actual stragglers and the Recall, i.e., percent of actual detected stragglers, is also relatively low at 56%. For the same case, the hierarchical approach (i.e., a green-driven detector based on the default one) achieves a Precision of 99% and a Recall of 29%. This increase in Precision can be translated to achieve lower execution time and energy consumption, and thus higher performance and energy efficiency; compared to the default Hadoop mechanism, the energy consumption is reduced by almost 31%. These results demonstrate how our framework can offer useful insights and be applied in practical settings to characterize and design new straggler detection mechanisms for MapReduce systems.

A Comprehensive View of Scheduling Algorithms for MapReduce Framework in Hadoop

We reside in information world which makes great difficulty in quantifying the aggregate volume of information put away electronically, and exist in the unit of Zettabytes or Exabyte, hence termed as Big Data. They can be unstructured, organized or semi organized, processing of these data cannot able to be done with normal data management techniques. Hadoop framework is utilized to process huge datasets in a proficient and economical way. MapReduce system is utilized to gather information according to the solicitation. To handle expansive volume of information legitimate planning is obliged to accomplish more noteworthy execution. Scheduling jobs in parallel across the nodes is an important concern in MapReduce clusters. Also to improve the performance with the large volume of data proper scheduling method should be adopted. The objective of the work is to study about the different scheduling techniques used in Hadoop environment and to achieve effective fairness without starvation.

MapReduce machine covering problem on a small number of machines

We study machine covering problem in MapReduce system. Each job consists of two sets of tasks, namely the map tasks and reduce tasks. A job’s reduce tasks can only be processed after all its map tasks are finished. The map tasks are fractional, i.e., they can be arbitrarily split and processed on different machines in parallel. Our goal is to maximize the minimum machine completion time. We consider two variants of the problem, namely the cases involving preemptive reduce tasks and non-preemptive reduce tasks. For preemptive reduce tasks, we present optimal solution algorithms for the problem on two and three machines. For non-preemptive reduce tasks, we provide an approximation algorithm with a tight worse-case ratio of $$\frac{4}{3}$$ for the problem on two machines.

Optimal online algorithms for MapReduce scheduling on two uniform machines

We study online scheduling on two uniform machines in the MapReduce system. Each job consists of two sets of tasks, namely the map tasks and reduce tasks. A job’s reduce tasks can only be processed after all its map tasks are finished. The map tasks are fractional, i.e., they can be arbitrarily split and processed on different machines in parallel. Our goal is to find a schedule that minimizes the makespan. We consider two variants of the problem, namely the cases involving preemptive reduce tasks and non-preemptive reduce tasks. We provide lower bounds for both variants. For preemptive reduce tasks, we present an optimal online algorithm with a competitive ratio of $$\frac{\sqrt{s^{2}+2s+5}+1-s}{2}$$ , where $$s\ge 1$$ is the ratio between the speeds of the two machines. For non-preemptive reduce tasks, we show that the $${ LS}$$ -like algorithm is optimal and its competitive ratio is $$\frac{2s+1}{s+1}$$ if $$s<\frac{1+\sqrt{5}}{2}$$ and $$\frac{s+1}{s}$$ if $$s\ge \frac{1+\sqrt{5}}{2}$$ .

CorrectMR: Authentication of Distributed SQL Execution on MapReduce

In this paper, we consider the SQL Selection-GroupBy-Aggregation (SGA) query evaluation on an untrusted MapReduce system in which mappers and reducers may return incorrect results. We design ${\sl CorrectMR}$ C o r r e c t M R , a system that supports efficient verification of result correctness for both intermediate and final results of SGA queries. ${\sl CorrectMR}$ C o r r e c t M R includes the design of Pedersen Merkle R-tree (PMR-tree), a new authenticated data structure (ADS). To enable efficient verification, ${\sl CorrectMR}$ C o r r e c t M R includes a distributed ADS construction mechanism that allows mappers/reducers to construct PMR-trees in parallel without a centralized party. ${\sl CorrectMR}$ C o r r e c t M R provides the following verification functionality: (1) correctness verification of PMR-trees by replication; (2) correctness verification of intermediate (final, resp.) query results by constructing local ( global , resp.) PMR-trees and verification objects. Our experimental results demonstrate the efficiency and effectiveness of ${\sl CorrectMR}$ C o r r e c t M R .

Map reduce technique for parallel-automata analysis of large scale rainfall data

Vast scale rainfall information assumes an imperative part in farming field thus early expectation of rainfall is important for the better finan-cial development of a nation. Rainfall expectation is an expert among the most troublesome issue far and wide in a year back. This data is generally secured in the unstructured course of action. Along these, tremendous measure of data has been accumulated and archived. Thus, storage and handling of such tremendous information for accurate rainfall forecast are a major test. Big Data innovation like Hadoop have developed to fathom the difficulties and issues of huge information utilizing distributed computing. Till date few examinations have been accounted for on the preparing of vast scale rainfall information utilizing MapReduce. In this paper, the huge scale rainfall information is anticipated by utilizing MapReduce system which plays out the capacities which are required and diminishes the task to get proficient ar-rangements through taking the information and isolating into smaller tasks. At that point, the three Regression Automata (RA) algorithms such as Linear Regression automata, Support Vector Regression Automata and Logistic Regression Automata. are utilized to forecast the future esteem of large scale rainfall data. The proposed framework serves as a tool that takes in the rainfall information from diminished information as input and predicts the future rainfall. The outcomes obviously demonstrate that the all the three RA models can anticipate the rainfall productively in different terms, such as, error rate, coefficients and mean square error.

Load balancing in join algorithms for skewed data in MapReduce systems

Join is an essential tool for data analysis which collected from different data sources. MapReduce has emerged as a prominent programming model for processing of massive data. However, traditional join algorithms based on MapReduce are not efficient when handling skewed data. The presence of data skew in input data leads to considerable load imbalance and performance degradation. This paper proposes a new skew-insensitive method, called fine-grained partitioning for skew data (FGSD) which can improve the load balancing for reduce tasks. The proposed method considers the properties of both input and output data through a proposed stream sampling algorithm. FGSD introduces a new approach for distribution of input data which leads to efficiently handling redistribution and join product skew. The experimental results confirm that our solution can not only achieve higher balancing performance, but also reduce the execution time of a job with varying degrees of the data skew. Furthermore, FGSD does not require any modification to the MapReduce environment and is applicable to complex join.