MapReduce Processing Research Articles

Online makespan minimization for MapReduce scheduling on multiple parallel machines

Abstract In this work, we investigate the online MapReduce processing problem on m m uniform parallel machines, aiming at minimizing the makespan. Each job consists of two sets of tasks, namely, the map tasks and the reduce tasks. A job’s map tasks can be arbitrarily split and processed on different machines simultaneously, while its reduce tasks can only be processed after all its map tasks have been completed. We assume that the reduce tasks are preemptive, but cannot be processed on different machines in parallel. We provide a new lower bound for this problem and present an online algorithm with a competitive ratio of 2 − 1 m 2-\frac{1}{m} ( m m is the number of machines) when the speeds of the machines are 1.

Distributed MapReduce processing of location-based aggregate queries in road networks

Distributed MapReduce processing of location-based aggregate queries in road networks

Improving Hadoop MapReduce performance on heterogeneous single board computer clusters

Over the past decade, Apache Hadoop has become a leading framework for big data processing. Single board computer (SBC) clusters, predominantly adopting Raspberry Pi (RPi), have been employed to explore the potential of MapReduce processing in terms of low power and cost because, capital costs aside, power consumption has also become a primary concern in many industries. After building SBC clusters, it is prevalent to consider adding more nodes, particularly newer generation SBCs, to the existing clusters or replacing old (or inactive) nodes with new ones to improve performance, inevitably causing heterogeneous SBC clusters. The Hadoop framework on these heterogeneous SBC clusters creates challenging new problems due to computing resource discrepancies in each node. Native Hadoop does not carefully consider the heterogeneity of the cluster nodes. Consequently, heterogeneous SBC Hadoop clusters result in significant performance variation or, more critically, persistent node failures. This paper proposes a new Hadoop Yet Another Resource Negotiator (YARN) architecture design to improve MapReduce performance on heterogeneous SBC Hadoop clusters with tight computing resources. We newly implement two main scheduling policies on Hadoop YARN based on the correct computing resource information that each SBC node provides: (1) two (master-driven vs. slave-driven) MapReduce task scheduling frameworks to determine more effective processing modes and (2) ApplicationMaster (AM) and reduce task distribution mechanisms to provide the best Hadoop performance by minimizing performance variation. Thus, the proposed Hadoop framework makes the best use of the performance-frugal SBC Hadoop cluster by intelligently distributing MapReduce tasks to each node. To our knowledge, the proposed framework is the first redesigned Hadoop YARN architecture to address various challenging problems particularly on heterogeneous SBC Hadoop clusters for big data processing. The extensive experiments with Hadoop benchmarks demonstrate that the redesigned framework performs better performance than the native Hadoop by an average of 2.55× and 1.55× under I/O intensive and CPU-intensive workloads, respectively.

A distributed attribute reduction based on neighborhood evidential conflict with Apache Spark

Attribute reduction is widely employed to improve the efficiency and accuracy of data analysis by eliminating redundant and irrelevant attributes from datasets. However, with the emergence of growing big data, the sequential execution of such algorithms becomes time-consuming and requires distributed computing capabilities to achieve scalable parallelization. This study proposes a novel attribute reduction algorithm for neighborhood decision systems. We introduce two novel metrics—the neighborhood evidential conflict degree (NECD) and neighborhood evidential conflict rate (NECR)—to compute heterogeneity between samples in the neighborhood and assess the significance of attributes in the feature space, respectively. These metrics assess the quality and selection of attribute subsets in attribute reduction, improving classification accuracy and computational efficiency. We also develop a sequentially forward selection attribute reduction method to select a feature subset through the defined NECR. Finally, we develop a distributed attribute reduction algorithm implemented in Apache Spark. Our approach involves a two-phase Map-Reduce process for K-Nearest Neighbors search, evidence combination, and NECR computation. NECR, as a measure of feature subset quality, enhances the feature subset's decision approximation capability of the data. Experimental results on small and large datasets demonstrate that the proposed algorithm outperforms benchmarking algorithms regarding classification accuracy and computational efficiency.

Using Hadoop, Analyze Airport Authority

A modern airport has access to a vast quantity of information, including the number of flights, arrival and departure times, flight routes, the number of airports that operate in each nation, a list of the current active airlines in each nation, etc. DBMS has so far been utilized to store and analyze the airport database. An airport database management system uses SQL to apply queries to all of the records. B-tree or distributed hash tables are used by My SQL to analyze the database. It is unable to analyze a large volume of data. Data is stored in a parallel database. This system's disadvantage is that it can't handle an increase in massive amounts of data to a significant degree. Therefore, the Hadoop database file system is employed to solve this issue. Hadoop uses HDFS and the Map-Reduce processing architecture to store vast amounts of data and provide the data in parallel. We made an effort to examine in-depth research of airline data sets, such as a list of airports in India, a list of airlines with no stops, a list of codeshare airlines by country with the most airports, and a list of active airlines in the United States. Here, we concentrated on processing large data sets in a distributed setting utilizing the hive component of the Hadoop ecosystem. The developers and business analysts will be helped by this effort when it comes to obtaining and handling their user requests

Hadoop Small Image Processing Technology Based on Big Data Processing and Its Application Effect in Face Feature Extraction and Face Recognition System Design

From standard software to crucial applications, face recognition (FR) is always at the core of many unique advances over the past 20 years. Big data is a rapidly growing collection of data. It is important in many applications such as medical, academic, and industries, and it refers to vast data sources that are hard to analyze, store, and interpret for subsequent procedures. Hadoop is an open-source large data processing platform that stores and analyses data in scalable computer clusters. FR technology is built on Hadoop processing to boost the effectiveness of recognition using MapReduce computing. This paper presents a novel adaptive fine-tuned AdaBoost (AFTA) algorithm to enhance the FR in the Hadoop processing. Collected face data sets are employed in preprocessing stage to normalize and enhance the superiority through the median filter (MF) and contour-based image enhancement (CIE), correspondingly. To handle vast quantities of the data, boosted k-means clustering (BKC) approach is used over the Hadoop servers for the MapReduce process. The binary partition tree (BPT) approach is employed in the segmentation stage to split the data into many subgroups. To lessen the dimension of the data, we use the Gabor filter. To select the consistent features, threshold-based binary particle swarm optimization (T-BPSO) is applied. Then, our proposed technique is utilized in the FR process. Finally, the performance metrics of the proposed technique are examined and compared with existing techniques to accomplish our research with the greatest effectiveness. For accuracy, precision, recall, and F-score, we attained the proposed (AFTA) values of 99.67, 97.13, 95.21, and 94.11, respectively. The outcomes are depicted in graphical representation by employing the MATLAB tool.

The Performance Optimization of Big Data Processing by Adaptive MapReduce Workflow

The discussion context of this paper is big data processing of MapReduce by volunteer computing in dynamic and opportunistic environments. This paper conducts a series of simulations to explore the relationship between the overall performance of volunteer overlays responding to different workload of big data problems. The discovery from the simulations includes some optimization points in overlay size, going over which by adding more volunteers brings little benefit for the overall performance. Based on the discovery of optimization points, this paper proposes a bootstrapping protocol, which can adapt volunteers into variable-sizes overlays, enabling workflow of single-round MapReduce or multiple-round MapReduce, and a single or multiple overlays for each round. The variable overlays aim to create adaptive workflow during MapReduce processing, so that the optimization points can be caught. As another benefit, the unnecessary computing-capacities can be released during computing when the optimization points are reached. The case study shows a few optimization workflows that are formed by the proposed bootstrapping protocol to process the big data cases. The workflows lead to the optimization points and dynamically balance the workload at the same time. The experiment results have demonstrated that the optimization strategies have either achieved 36% or 71% higher performance than the plain MapReduce workflow and minimized the use of computing resources by releasing 12.5% to 75% volunteers during computing, where the original plain MapReduce must hold all the volunteers to the end of computing. The extensibility of the simulation parameterization to more diverse real-world applications have been clarified.

Enhancing Leakage Prevention for MapReduce

When public clouds become the platform of choice for MapReduce processing, users are placing higher demands on the privacy of the job data and program. A number of solutions employed trusted hardware to protect MapReduce tasks. However, existing works pointed out that simply protecting individual nodes in the MapReduce cluster with trusted hardware and protecting cross-node communication with encryption still leak information from side-channels. Specifically, attackers can derive data information by observing and manipulating cross-node communication traffic volumes. Although existing works proposed some solutions to prevent such leakage, in this paper, we show that previous solutions still leak critical job information. Additionally, our study shows that previous solutions have limitations from other aspects, including data restriction, partition function restriction, reliability issue, and high overheads. To address all the discovered limitations, we introduced the Strong Shuffle solution. Our analysis and experimental results showed that our solution has reduced the information leakage and addressed other discovered limitations. To support Strong Shuffle, we proposed a variant Bloom Filter, named Group-based Dynamic Bloom Filter (GDBF). Our theoretical analysis showed that GDBF has lower performance and storage overhead than the traditional Scalable Bloom Filter.

Object Detection in Remote Sensing Images Using Picture Fuzzy Clustering and MapReduce

In image processing, one of the most important steps is image segmentation. The objects in remote sensing images often have to be detected in order to perform next steps in image processing. Remote sensing images usually have large size and various spatial resolutions. Thus, detecting objects in remote sensing images is very complicated. In this paper, we develop a model to detect objects in remote sensing images based on the combination of picture fuzzy clustering and MapReduce method (denoted as MPFC). Firstly, picture fuzzy clustering is applied to segment the input images. Then, MapReduce is used to reduce the runtime with the guarantee of quality. To convert data for MapReduce processing, two new procedures are introduced, including Map_PFC and Reduce_PFC. The formal representation and details of two these procedures are presented in this paper. The experiments on satellite image and remote sensing image datasets are given to evaluate proposed model. Validity indices and time consuming are used to compare proposed model to picture fuzzy clustering model. The values of validity indices show that picture fuzzy clustering integrated to MapReduce gets better quality of segmentation than using picture fuzzy clustering only. Moreover, on two selected image datasets, the run time of MPFC model is much less than that of picture fuzzy clustering.

Retraction Note to: Effect of ecological restoration in mountainous area based on C4.5 algorithm of decision tree and college English distance learning system

In this paper, we first consider the phenomenon of exponential growth of existing equipment data in substation, introduce the cloud computing operation mode, and build a more scalable MapReduce processing system based on Hadoop platform. Cloud computing is completed by using decision tree C4.5 algorithm, and the algorithm is used to optimize the parallel computing mode on the Hadoop platform as the core of the same data processing. Based on this point, this paper considers the impact of the ecological restoration effect of L County in the northern mountainous area of T city on the environment of the study area, analyzes the climate and hydrological conditions of the study area, provides the basic data of the mountain ecological environment of Y mountain area for the study of mountain landscape planning, and facilitates the practice. The combination of landscape planning, ecological restoration, and environmental reconstruction system theory provides a theoretical and practical basis for ecological planning in the central mountainous area of T city. Finally, based on MOOC components and software architecture, this paper constructs a college English remote system. After comprehensive testing in many aspects, this paper reduces the cost of software development in this field, which not only improves the reliability of the system, but also shortens the development cycle, speeds up the maintenance speed of the system, and increases the scale of the system. About the system testing, this paper tests the function, performance, feasibility, integrity, and security of the system through the network testing method and cookie method. Based on the basic C4.5 decision-making algorithm and college English remote system, this paper studies the impact of mountain ecological environment restoration, provides ideas for promoting the development of the system, and provides reference for the reform and practice methods and processes in other courses.

Triple DES-WSA: an efficient security algorithm providing integrity with aid of MapReduce process

In the last decade, major complications of big-data-to-IoT are data-management, security and time-management and thus the objective of the research is recognising with the problems in big data to extend IoT. In order to accord with the data-management problem an ontology-based matching using H matching algorithm is implemented subsequent to pillar k-means clustering algorithm that segregates meaningful and meaningless information. The time management problem is recognised with the aid of Hadoop with MapReduce that specifies the location hitherto. Consequently, triple DES algorithm with WPA is utilised and that is accountable for providing security by the usage of three keys for both the encryption-decryption. On the other hand, WPA is employed to the data integrity to substantiate whether the data is altered. We have applied ontology based on H-matching algorithm proposed in the sequence of steps: the stop word removal, stemming process, grouping process and the ontology making for matching. We have proposed the methodology to handle three parameters: confidentiality, integrity and authenticity in the context of big-data.

Big Data analysis of demand side management for Industrial IOT applications

Companies today demand even more high quality services and infrastructure in the field ofIndustrial Internet IOT. There are several problems facing smart urbanization. Of special concern are secured energy demand side management. The IIoT makes ransomware, cyber attacks and other security threats open to industrial systems. IIoT will provide meaningful solutions to these problems through the amalgamation of big data analytics. That first author recommends a safe, reliable Demand Side Managementengine for a smart cooperative client using IIoT based Big Data Analysis. A highly distributed framework to the theoretical engine for optimizing DSM across a Home Area Network. A payload authentication framework based on a lightweight handshake mechanism is used to improve the protection of this engine. In order to make users to control different server native resources energy efficiently, our theoretical method uses the portable utilizes of the Constrained Application Protocol. Moreover, data streams with parallel MapReduce processing are managed using big data analysis. In addition, we use Apache Spark with Apache Hadoop to check the theoretical engine to test our input transform match on stable datafile. The test results indicate that the architecture outlined provides useful insights into the intelligent social communities of the IIoT.

Power-Law Distributed Graph Generation With MapReduce

A graph generator is a tool which allows to create graph-like data whose structural properties are very similar to those found in real world networks. This paper presents two methods to generate graphs with power-law edge distribution based on the MapReduce processing model that can be easily implemented to run on top of Apache Hadoop. The proposed methods allow the generation of directed and undirected power-law distributed graphs without repeated edges. Our experimental evaluation shows that our methods are efficient and scalable in terms of both graph size and cluster capacity.

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

With the rise of big data, more and more users will launch computing systems to process a large volume of data in various applications. A Scheduling algorithm is crucial to the performance of the processing platforms, especially when they are concurrently executing a batch of jobs. Such jobs usually represent multiple stages. Each stage produces the intermediate data which will be piped to the next stage for further processing. However, the scheduling problem in a big data computing system is different from the traditional multi-stage job scheduling problem as for any two consecutive stages, the later stage usually starts before the former stage is finished to “shuffle” the intermediate data. In this paper, we consider MapReduce/Hadoop as a representative computing system and develop a new strategy named OMO, Optimize MapReduce Overlap with a Good Start (Reduce) and a Good Finish (Map). A MapReduce job contains two consecutive phases: map and reduce. Our general target is to optimize the internal overlap between these two phases. There are two new techniques included in our solution, Lazy start of reduce tasks and Batch finish of map tasks, which aim to approach an effective alignment of the two phases based on the characteristics of the MapReduce process. OMO has been implemented on the Hadoop system with extensive experiments for performance evaluation. The results show that OMO's performance is superior in terms of total completion time (i.e., makespan) of a batch of jobs.

Triple DES-WSA: an efficient security algorithm providing integrity with aid of MapReduce process

Triple DES-WSA: an efficient security algorithm providing integrity with aid of MapReduce process

Fault diagnosis method of submersible screw pump based on random forest

The difficulty in directly determining the failure mode of the submersible screw pump will shorten the life of the system and the normal production of the oil well. This thesis aims to identify the fault forms of submersible screw pump accurately and efficiently, and proposes a fault diagnosis method of the submersible screw pump based on random forest. HDFS storage system and MapReduce processing system are established based on Hadoop big data processing platform; Furthermore, the Bagging algorithm is used to collect the training set data. Also, this thesis adopts the CART method to establish the sample library and the decision trees for a random forest model. Six continuous variables, four categorical variables and fault categories of submersible screw pump oil production system are used for training the decision trees. As several decision trees constitute a random forest model, the parameters to be tested are input into the random forest models, and various types of decision trees are used to determine the failure category in the submersible screw pump. It has been verified that the accuracy rate of fault diagnosis is 92.86%. This thesis can provide some meaningful guidance for timely detection of the causes of downhole unit failures, reducing oil well production losses, and accelerating the promotion and application of submersible screw pumps in oil fields.

Cross-MapReduce: Data transfer reduction in geo-distributed MapReduce

The MapReduce model is widely used to store and process big data in a distributed manner. MapReduce was originally developed for a single tightly coupled cluster of computers. Approaches such as Hierarchical and Geo-Hadoop are designed to address geo-distributed MapReduce processing. However, these methods still suffer from high inter-cluster data transfer over the Internet, which is prohibitive for processing today’s globally big data. In line with our thinking that there is no need to transfer the entire intermediate results to a single global reducer, we propose Cross-MapReduce, a framework for geo-distributed MapReduce processing. Before any massive data transfer, our proposed method finds a set of best global reducers to minimize transferred data volumes. We propose a graph called Global Reduction Graph (GRG) to determine the number and the locations of the global reducers. We conducted extensive experimental evaluations using a real testbed to demonstrate the effectiveness of Cross-MapReduce. The experimental results show that Cross-MapReduce significantly outperforms the Hierarchical and Geo-Hadoop approaches and reduces the amount of data transfer over the Internet by 40%.

The File System Recommendations to Reduce the Space and Time Parameters in Hadoop File Storage and Map Reduce Processing of Big Data Applications

The study of Hadoop Distributed File System (HDFS) and Map Reduce (MR) are the key aspects of the Hadoop framework. The big data scenarios like Face Book (FB) data processing or the twitter analytics such as storing the tweets and processing the tweets is other scenario of big data which can depends on Hadoop framework to perform the storage and processing through which further analytics can be done. The point here is the usage of space and time in the processing of the above-mentioned huge amounts of the data definitely leads to higher amounts of space and time consumption of the Hadoop framework. The problem here is usage of huge amounts of the space and at the same time the processing time is also high which need to be reduced so as to get the fastest response from the framework. The attempt is important as all the other eco system tools also depends on HDFS and MR so as to perform the data storage and processing of the data and alternative architecture so as to improve the usage of the space and effective utilization of the resources so as to reduce the time requirements of the framework. The outcome of the work is faster data processing and less space utilization of the framework in the processing of MR along with other eco system tools like Hive, Flume, Sqoop and Pig Latin. The work is proposing an alternative framework of the HDFS and MR and the name we are assigning is Unified Space Allocation and Data Processing with Metadata based Distributed File System (USAMDFS).

RETRACTED ARTICLE: Quantitative evaluation method of cloud resources based on work scheduling

With the appearance of the period of large information, the customary information preparing model and asset the executive’s model have been not able to fulfill the developing need. In this way, the recreation of the quantitative assessment strategy for cloud asset dependent on MapReduce figuring mode was advanced right now. The quantitative assessment model and record foundation of cloud assets in Hadoop stage and MapReduce processing mode were abridged. Through the quantitative assessment strategy for cloud assets dependent on MapReduce processing mode, the information was gathered and examined. The outcomes show that the quantitative assessment technique for cloud asset dependent on MapReduce processing model has a decent figuring force and the capacity to anticipate the utilization of cloud assets.

Verifying Properties of MapReduce-Based Big Data Processing

Big data techniques are widely used in various fields. To deal with large data sets efficiently, a new programming framework MapReduce has emerged. Thus, new verification challenges arise to improve the reliability of big data processing. In this article, MapReduce processes are implemented by modeling simulation and verification language programs. Then, several data properties such as data soundness, nonconflict, nonduplication, cooperation, and completeness are taken into account. Moreover, these properties are specified by propositional projection temporal logic formulas. To verify these properties, a runtime verification approach at code level based on unified model checking is employed. In addition, two case studies are conducted to demonstrate our approach: sparse matrix multiplication and tracking down suspected patients of an infectious disease.