Yet Another Resource Negotiator Research Articles

Improving Hadoop MapReduce performance on heterogeneous single board computer clusters

Over the past decade, Apache Hadoop has become a leading framework for big data processing. Single board computer (SBC) clusters, predominantly adopting Raspberry Pi (RPi), have been employed to explore the potential of MapReduce processing in terms of low power and cost because, capital costs aside, power consumption has also become a primary concern in many industries. After building SBC clusters, it is prevalent to consider adding more nodes, particularly newer generation SBCs, to the existing clusters or replacing old (or inactive) nodes with new ones to improve performance, inevitably causing heterogeneous SBC clusters. The Hadoop framework on these heterogeneous SBC clusters creates challenging new problems due to computing resource discrepancies in each node. Native Hadoop does not carefully consider the heterogeneity of the cluster nodes. Consequently, heterogeneous SBC Hadoop clusters result in significant performance variation or, more critically, persistent node failures. This paper proposes a new Hadoop Yet Another Resource Negotiator (YARN) architecture design to improve MapReduce performance on heterogeneous SBC Hadoop clusters with tight computing resources. We newly implement two main scheduling policies on Hadoop YARN based on the correct computing resource information that each SBC node provides: (1) two (master-driven vs. slave-driven) MapReduce task scheduling frameworks to determine more effective processing modes and (2) ApplicationMaster (AM) and reduce task distribution mechanisms to provide the best Hadoop performance by minimizing performance variation. Thus, the proposed Hadoop framework makes the best use of the performance-frugal SBC Hadoop cluster by intelligently distributing MapReduce tasks to each node. To our knowledge, the proposed framework is the first redesigned Hadoop YARN architecture to address various challenging problems particularly on heterogeneous SBC Hadoop clusters for big data processing. The extensive experiments with Hadoop benchmarks demonstrate that the redesigned framework performs better performance than the native Hadoop by an average of 2.55× and 1.55× under I/O intensive and CPU-intensive workloads, respectively.

An Optimized IoT-enabled Big Data Analytics Architecture for Edge-Cloud Computing.

The awareness of edge computing is attaining eminence and is largely acknowledged with the rise of Internet of Things (IoT). Edge-enabled solutions offer efficient computing and control at the network edge to resolve the scalability and latency-related concerns. Though, it comes to be challenging for edge computing to tackle diverse applications of IoT as they produce massive heterogeneous data. The IoT-enabled frameworks for Big Data analytics face numerous challenges in their existing structural design, for instance, the high volume of data storage and processing, data heterogeneity, and processing time among others. Moreover, the existing proposals lack effective parallel data loading and robust mechanisms for handling communication overhead. To address these challenges, we propose an optimized IoT-enabled big data analytics architecture for edge-cloud computing using machine learning. In the proposed scheme, an edge intelligence module is introduced to process and store the big data efficiently at the edges of the network with the integration of cloud technology. The proposed scheme is composed of two layers: IoT-edge and Cloud-processing. The data injection and storage is carried out with an optimized MapReduce parallel algorithm. Optimized Yet Another Resource Negotiator (YARN) is used for efficiently managing the cluster. The proposed data design is experimentally simulated with an authentic dataset using Apache Spark. The comparative analysis is decorated with existing proposals and traditional mechanisms. The results justify the efficiency of our proposed work.

Geometric-thermal error control system for gear profile grinding machine

Geometric and thermal errors, which are the main error factors for reducing the machining accuracy, should be controlled. But the control effect is poor, which is a stumbling block to limit the wide application of the error control. In this study, a geometric-thermal error control system (GTECS) is designed for gear profile grinding machines. For the mist layer of GTECS, the wireless sensor network is designed to realize the data collection and transfer. For the edge layer of GTECS, the edge controller is designed to conduct the sensitive error analysis. For the fog layer, the control module is designed to conduct the geometric and thermal error prediction. In this layer, the analytical model of the rolling guide/slider system is proposed to calculate geometric errors of X- and Z- axes, and the thermal boundary conditions are calculated, and the thermal error models of the spindle and C-axis are proposed based on transfer learning model (TLM) of the sooty tern optimization (STO)-bilinear temporal convolutional network (BTCN). For the cloud layer, the data computation and management are realized by Hadoop and Yet Another Resource Negotiator (YARN), respectively. The geometric and thermal error models of X- and Z-axes, thermal errors models of the spindle and C-axis, and multi-source error model are embedded into it. With the execution of GTECS, the geometric precision for the total tooth profile deviation and tooth profile deviation are increased from ISO level 8 to ISO level 5 and from ISO level 5 to ISO level 3, respectively.

Smart teledentistry healthcare architecture for medical big data analysis using IoT-enabled environment

The current spread out in Big Data analytics and the medical Internet of Things (IoT) originated the recognition of smart health. Smart health is the integration of devices, sensors, cameras, and objects or things embedded with sensors that generate an enormous amount of data known as big data. IoT-enabled smart health management assimilates the digital traces generated by things and humans including teledentistry. The key purpose of teledentistry is to analyze the data and produce valuable insights and hidden patterns to offer services and overcome the existing challenges. This article proposes a data management design using Big Data analytics for smart teledentistry planning. The proposed scheme is the customization of the present parallel platforms to accomplish effective processing. The design offers three different modules which are the data pre-processing, data ingestion, and data processing module. The pre-processing is performed to speed up the real data processing. The recent schemes lack well-organized and effective parallel data loading and proficient communication. The Big Data storage and ingestion are achieved using optimized utility and parallel mechanisms. Optimized RDD-enabled (resilient distribution) Yet Another Resource Negotiator (YARN) based proposal is designed for resourceful cluster administration and computation of medical Big Data. The proposed data design is experimentally simulated with authentic datasets using Apache Spark 3.0 framework. The proposed architecture is compared with existing state-of-the-art proposals. The simulation results reveal the efficacy of the proposed system.

Apache Hadoop-MapReduce on YARN framework latency

Abstract Big Data is currently a fertile field for researchers and scientific companies around the world, due to the emergence of new technologies, Internet of Things (IoT) and means of communication such as social networking sites, which has led to a notable increase in the amount of data produced each day. In this paper, we discuss the challenges of Big Data and we are going to present the famous Apache Hadoop framework and we are going to test the latency at the level of execution of a MapReduce job based on YARN (Yet Another Resource Negotiator) by applying a Java wordcount application on different text files which resumes the sales of a store, which has prepared beforehand, finally going to make a synthesis of the results.

Fine-tuning Resource Allocation of Apache Spark Distributed Multinode Cluster for Faster Processing of Network-trace Data

In the field of network security, the task of process-ing and analyzing huge amount of Packet CAPture (PCAP) data is of utmost importance for developing and monitoring the behavior of networks, having an intrusion detection and prevention system, firewall etc. In recent times, Apache Spark in combination with Hadoop Yet-Another-Resource-Negotiator (YARN) is evolving as a generic Big Data processing platform. While processing raw network packets, timely inference of network security is a primitive requirement. However, to the best of our knowledge, no prior work has focused on systematic study on fine-tuning the resources, scalability and performance of distributed Apache Spark cluster (while processing PCAP data). For obtaining best performance, various cluster parameters like number of cluster nodes, number of cores utilized from each node, total number of executors run in the cluster, amount of main-memory used from each node, executor memory overhead allotted for each node to handle garbage collection issue, etc., have been fine-tuned, which is the focus of the proposed work. Through the proposed strategy, we could analyze 85GB of data (provided by CSIR Fourth Paradigm Institute) in just 78 seconds, using 32 node (256 cores) Spark cluster. This would otherwise take around 30 minutes in traditional processing systems.

A New Data Layout Scheme for Energy-Efficient MapReduce Processing Tasks

Yet Another Resource Negotiator (YARN) is a framework to manage and allocate resource requests from applications that process big data stored in HDFS. However, dynamic power management methods are not efficient when YARN manage applications to process big data stored in the default data layout of HDFS. In this paper, we propose a new data layout scheme that can be implemented for HDFS. A comparison between our proposal and the existing HDFS data layout scheme shows that the new data layout algorithm significantly reduces the energy consumption at the slight expense of the mean response time of jobs.

Determination of Business Intelligent Using Micro Financial Analysis of Tamilnadu SME

Financial data processing is quite obvious where per mean time the collection of data is more for business intelligent using Information and Communication Technology. Most people in urban areas financially face problems which turns their day to day life into tragedy. Business intelligence tries to budget different kinds of plans, implementing to our beloved citizens for their comfort over state government. Even though, the prediction process fails in various manners while processing large amount of dataset ends with latency and cost consumption. To overcome financial data analysis among huge data feeds, big data hadoop is required for categorized result under various sectors. In the paper proposed a YARN (Yet Another Resource Negotiator) mechanism of hadoop framework to analyze micro small and medium enterprises data of Tamilnadu financial analysis. It implements hive tool to store data in Hadoop Distributed File System (HDFS), import analysis and produce result among them. Whereas hadoop/hive uses map and reduce to compute huge datasets, tables with structured data, CSV files, etc., each cluster perform multiple mapper and reducer jobs allocated for their categorized results, and the framework effectively accumulates as well as increase in performance via parallel processing with the help of multiple map reduce method. Based on Experimental estimations, proposed YARN framework enhances the less processing time 12 minutes, high Data Size 2 GB and accuracy 22.26% of the proposed framework contrasted than previous frameworks.

Big-Data Mechanism and Energy-Policy Design

Investigation is building a digital framework stage to help multivariate representation of information gathered from conveyed sensor arrange. three new strategies were presented in this stage: First, a half breed information reserving technique that takes favorable circumstances of an adaptable and appropriated time arrangement database, open tSDB, to acknowledge productive information recovery. Second, a hyper-dimensional information 3D shape is set up to delineate interpret multivariate and heterogeneous sensor information into a typical information structure to help area mindful visual examination. third, an information driven representation module is executed to help intelligent and dynamic perception on a recreated virtual globe. A Yet Another Resource Negotiator (YARN) leads to show the great time execution of the proposed framework. The appearance of cutting edge sequencing stages, which have significantly expanded the extent of small scale biomedical-related tasks, a few superior grouping examination pipelines are currently accessible to agents for smaller scale biomedical investigation. to anticipate the work will make an important commitment to both the representation building piece improvement in digital foundation examine and the headway of visual introduction and investigation of sensor information in area sciences.

Application of feedback control on the cloud-based web simulation system

This article illustrated the development of a web simulation system based on Matlab distributed parallel cluster. The web simulation system construct a visual and controllable simulation interface using javascript and Asynchronous JavaScript and XML(AJAX) technology. In order to realize the Matlab resources dynamic allocation and management, elastic resources management method has been adopted. To achieve the multi-user concurrent access, the resource management system has been designed by adopting principle of feedback control and principle of Apache Hadoop Yet Another Resource Negotiator(YARN). The feasibility of this system demonstrated through application of simulation and practical use.

In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model

MapReduce has been widely used in Hadoop for parallel processing larger-scale data for the last decade. However, remote-sensing (RS) algorithms based on the programming model are trapped in dense disk I/O operations and unconstrained network communication, and thus inappropriate for timely processing and analyzing massive, heterogeneous RS data. In this paper, a novel in-memory computing framework called Apache Spark (Spark) is introduced. Through its merits of transferring transformation to in-memory datasets of Spark, the shortages are eliminated. To facilitate implementation and assure high performance of Spark-based algorithms in a complex cloud computing environment, a strip-oriented parallel programming model is proposed. By incorporating strips of RS data with resilient distributed datasets (RDDs) of Spark, all-level parallel RS algorithms can be easily expressed with coarse-grained transformation primitives and BitTorrent-enabled broadcast variables. Additionally, a generic image partition method for Spark-based RS algorithms to efficiently generate differentiable key/value strips from a Hadoop distributed file system (HDFS) is implemented for concealing the heterogeneousness of RS data. Data-intensive multitasking algorithms and iteration-intensive algorithms were evaluated on a Hadoop yet another resource negotiator (YARN) platform. Experiments indicated that our Spark-based parallel algorithms are of great efficiency, a multitasking algorithm took less than 4 h to process more than half a terabyte of RS data on a small YARN cluster, and 9*9 convolution operations against a 909-MB image took less than 260 s. Further, the efficiency of iteration-intensive algorithms is insensitive to image size.

Performance evaluation of job schedulers on Hadoop YARN

SummaryTo solve the limitation of Hadoop on scalability, resource sharing, and application support, the open‐source community proposes the next generation of Hadoop's compute platform called Yet Another Resource Negotiator (YARN) by separating resource management functions from the programming model. This separation enables various application types to run on YARN in parallel. To achieve fair resource sharing and high resource utilization, YARN provides the capacity scheduler and the fair scheduler. However, the performance impacts of the two schedulers are not clear when mixed applications run on a YARN cluster. Therefore, in this paper, we study four scheduling‐policy combinations (SPCs for short) derived from the two schedulers and then evaluate the four SPCs in extensive scenarios, which consider not only four application types, but also three different queue structures for organizing applications. The experimental results enable YARN managers to comprehend the influences of different SPCs and different queue structures on mixed applications. The results also help them to select a proper SPC and an appropriate queue structure to achieve better application execution performance. Copyright © 2016 John Wiley & Sons, Ltd.

Advanced resource management with access control for multitenant Hadoop

Multitenancy has gained growing importance with the development and evolution of cloud computing technology. In a multitenant environment, multiple tenants with different demands can share a variety of computing resources (e.g., CPU, memory, storage, network, and data) within a single system, while each tenant remains logically isolated. This useful multitenancy concept offers highly efficient, and cost-effective systems without wasting computing resources to enterprises requiring similar environments for data processing and management. In this paper, we propose a novel approach supporting multitenancy features for Apache Hadoop, a large scale distributed system commonly used for processing big data. We first analyze the Hadoop framework focusing on yet another resource negotiator (YARN), which is responsible for managing resources, application runtime, and access control in the latest version of Hadoop. We then define the problems for supporting multitenancy and formally derive the requirements to solve these problems. Based on these requirements, we design the details of multitenant Hadoop. We also present experimental results to validate the data access control and to evaluate the performance enhancement of multitenant Hadoop.

GeoProcessing Workflow Models for Distributed Processing Frameworks

Information Systems (GIS) platforms are used to implement and deal with the massive spatial data, especially image data. Therefore, these platforms require high storage capacity and high computational power to process them. This paper aims of processing of Geodata using Distributed processing Frameworks. Large volume of Geodata cannot be processed using desktop GIS tools such as QGIS, ArcGIS, GRASS, OpenJUMP etc. Therefore, to handle and process on these types of large data, use of Hadoop Distributed processing framework needs to be deployed. GeoProcessing is a GIS operation used to manipulate spatial data. It is one of the original proposal in GIS development. Almost every GIS application is represented by a GeoProcessing Workflow. This paper explains the GeoProcessing Workflow for processing of image data. Also explains Hadoop Distributed File System (HDFS), MapReduce Programming Model and Yet Another Resource Negotiator (YARN) architecture, useful in large spatial data handling and analysis at fast rate.

Big Data Analysis with Dataset Scaling in Yet Another Resource Negotiator (YARN)

data is exceedingly large day by day. In some organizations, there is a need to analyze and process the gigantic data. This is a big data problem often faced by these organizations. It is not possible for single machine to handle that data. So we have used Apache Hadoop Distributed File System (HDFS) for storage and analysis. This paper shows experimental work done on the MapReduce Application on Health sector dataset. The result shows the behavior of the MapReduce application framework to map and reduce the big volume of the data. The main problem is to check the behavior of the MapReduce applications by increasing the size of dataset. Our analysis lies in understanding the Apache MapReduce application performance. We expect that execution time increases linearly with the dataset size but our analysis shows sometimes the execution time varies non- linearly with the increase in the dataset size. The experimental result shows that with scaling the datasets execution time distinguishes. KeywordsData, Hadoop, MapReduce, YARN, Single Node, Multi

Big Data 2.0: Cataclysm or Catalyst?

Big Data 2.0: Cataclysm or Catalyst?