Hadoop Cluster Research Articles

TCP proactive congestion control for east-west traffic: The marking threshold

Various extensions of TCP/IP have been proposed to reduce network latency; examples include Explicit Congestion Notification (ECN), Data Center TCP (DCTCP) and several proposals for Active Queue Management (AQM). Combining these techniques requires adjusting various parameters, and recent studies have found that it is difficult to do so while obtaining both high throughput performance and low latency. This is especially true for mixed use data centres that host both latency-sensitive applications and high-throughput workloads with east–west traffic such as Hadoop.This paper studies the difficulty in configuration, and characterises the problem as related to ACK packets. Such packets cannot be set as ECN Capable Transport (ECT), with the consequence that a disproportionate number of them are dropped. The same issue can affect other non-ECT-capable traffic that may co-exist on the network. We explain how this behavior adversely affects throughput, and propose a small change to the way that non-ECT-capable packets are handled in the network switches. Using NS–2 simulation, we demonstrate robust performance for modified AQMs on a Hadoop cluster, maintaining full throughput while reducing latency by 85%. We also demonstrate that commodity switches with shallow buffers are able to reach the same throughput as deeper buffer switches.Finally, we explain how both TCP using ECN and DCTCP can achieve the best performance using a simple marking threshold, in contrast to the current preference for relying on AQMs to mark packets. Overall, we provide recommendations to network equipment manufacturers, cluster administrators and the whole industry on how best to combine high-throughput and latency-sensitive workloads. This article is an extension of our previous work [1], which was published in Proceedings of the 19th IEEE International Conference on Cluster Computing (CLUSTER 2017).

A novel entropy-based dynamic data placement strategy for data intensive applications in Hadoop clusters

In the last decade, efficient data analysis of data-intensive applications has become an important research issue. Hadoop is the most widely used platform for data intensive application. However, majority of data placement strategies attempt placing related-data close to each other for faster access without considering new datasets, generated or for different MapReduce jobs. This paper deals with improving the map-reduce performance over multi-cluster datasets by means of a novel-entropy-based data placement strategy (EDPS) in three-phases. K-means clustering strategy is employed to extract dependencies among different datasets and group them into data-groups. Then these data-groups are placed in different datacenters while considering heterogeneity. Finally, an entropy-based grouping of the newly generated datasets where these datasets are grouped with most similar existing cluster based on their relative entropy. The experimental results show efficacy of the proposed three-fold dynamic grouping and data placement policy, which significantly reduces the time and improve Hadoop performance.

A Data Structure Perspective to the RDD-based Apriori Algorithm on Spark

During the recent years, a number of efficient and scalable frequent itemset mining algorithms for big data analytics have been proposed by many researchers. Initially, MapReduce-based frequent itemset mining algorithms on Hadoop cluster were proposed. Although, Hadoop has been developed as a cluster computing system for handling and processing big data, but the performance of Hadoop does not meet the expectation for the iterative algorithms of data mining, due to its high I/O, and writing and then reading intermediate results in the disk. Consequently, Spark has been developed as another cluster computing infrastructure which is much faster than Hadoop due to its in-memory computation. It is highly suitable for iterative algorithms and supports batch, interactive, iterative, and stream processing of data. Many frequent itemset mining algorithms have been re-designed on the Spark, and most of them are Apriori-based. All these Spark-based Apriori algorithms use Hash Tree as the underlying data structure. This paper investigates the efficiency of various data structures for the Spark-based Apriori. Although, the data structure perspective has been investigated previously, but for MapReduce-based Apriori, and it must be re-investigated in the distributed computing environment of Spark. The considered underlying data structures are Hash Tree, Trie, and Hash Table Trie. The experimental results on the benchmark datasets show that the performance of Spark-based Apriori with Trie and Hash Table Trie are almost similar but both perform many times better than Hash Tree in the distributed computing environment of Spark.

A novel entropy-based dynamic data placement strategy for data intensive applications in Hadoop clusters

In the last decade, efficient data analysis of data-intensive applications has become an important research issue. Hadoop is the most widely used platform for data intensive application. However, majority of data placement strategies attempt placing related-data close to each other for faster access without considering new datasets, generated or for different MapReduce jobs. This paper deals with improving the map-reduce performance over multi-cluster datasets by means of a novel-entropy-based data placement strategy (EDPS) in three-phases. K-means clustering strategy is employed to extract dependencies among different datasets and group them into data-groups. Then these data-groups are placed in different datacenters while considering heterogeneity. Finally, an entropy-based grouping of the newly generated datasets where these datasets are grouped with most similar existing cluster based on their relative entropy. The experimental results show efficacy of the proposed three-fold dynamic grouping and data placement policy, which significantly reduces the time and improve Hadoop performance.

An improved content splitting and merging algorithm for Hadoop clusters using component analysis and hamming distance

Distributed storage and processing of dataset of big data have become an integrated component of data science. With the technology progress towards the Internet of Things (IoTs), big data becomes more important. Therefore, processing of such data needs utmost concern for the ease of availability and accuracy. Various research has been executed till date for the efficient use of splitting and merging of content in the processing of data. But, somehow they lack in the generation of proper clusters in Hadoop. In this paper, we have shown an efficient approach of using splitting and merging process of data processing. We have used component analysis and hamming distance to generate thee clusters depending on the split values which is novel in this domain of work. The experimented results of our proposed approach provide better efficiency in term of discrete clusters and time consumption.

Towards an Energy Efficient Computing With Coordinated Performance-Aware Scheduling in Large Scale Data Clusters

Many prior works have investigated on how to increase the job processing performance and energy efficient computing in large scale clusters. However, they employ serialized scheduling approaches encompassed with task straggler “hunting” techniques which launches speculative tasks after detecting slow tasks. These slow tasks are detected through node instrumentation which collects system level information whilst tracking the task execution progress. Such approaches are however detrimental towards achieving maximum processing performance and preserving cluster energy as they increase communication overheads. In this paper, we observe that node instrumentation and serialized scheduling in existing works does not only degrade the job processing performance, but also increase cluster energy consumption. To alleviate this, we propose EPPADS, a light-weight scheduler which eradicates the need for instrumentation modules for job scheduling purposes. EPPADS schedules tasks in two stages, the slow-start phase (SSP) and accelerate phase (AccP). The SSP schedules initial tasks in the queue using baseline FIFO scheduling and records the initial execution times of the processing nodes, whilst tagging the effective and straggling nodes. The AccP uses the initial execution times to compute the processing nodes task distribution ratio of remaining tasks and schedules them in parallel using a single scheduling I/O, boosting up the processing performance. To amortize the computing energy costs, EPPADS implements a power management module that coordinates with the scheduling module and leverage on node tagging information, to place nodes in two different power transition pools, i.e., high and low state power pools. A single power transition signal per pool is then broadcasted to lower or raise the energy state in the low-power state pool and high-power state pool. Our evaluation using a Hadoop cluster shows that EPPADS achieves 30% and 22% performance improvement and 15% to 20% energy savings as compared to the FIFO and DynMon schedulers, respectively.

An improved content splitting and merging algorithm for Hadoop clusters using component analysis and hamming distance

Distributed storage and processing of dataset of big data have become an integrated component of data science. With the technology progress towards the Internet of Things (IoTs), big data becomes more important. Therefore, processing of such data needs utmost concern for the ease of availability and accuracy. Various research has been executed till date for the efficient use of splitting and merging of content in the processing of data. But, somehow they lack in the generation of proper clusters in Hadoop. In this paper, we have shown an efficient approach of using splitting and merging process of data processing. We have used component analysis and hamming distance to generate thee clusters depending on the split values which is novel in this domain of work. The experimented results of our proposed approach provide better efficiency in term of discrete clusters and time consumption.

Software Abstractions for Large-Scale Deep Learning Models in Big Data Analytics

The goal of big data analytics is to analyze datasets with a higher amount of volume, velocity, and variety for large-scale business intelligence problems. These workloads are normally processed with the distribution on massively parallel analytical systems. Deep learning is part of a broader family of machine learning methods based on learning representations of data. Deep learning plays a significant role in the information analysis by adding value to the massive amount of unsupervised data. A core domain of research is related to the development of deep learning algorithms for auto-extraction of complex data formats at a higher level of abstraction using the massive volumes of data. In this paper, we present the latest research trends in the development of parallel algorithms, optimization techniques, tools and libraries related to big data analytics and deep learning on various parallel architectures. The basic building blocks for deep learning such as Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) are identified and analyzed for parallelization of deep learning models. We proposed a parallel software API based on PyTorch, Hadoop Distributed File System (HDFS), Apache Hadoop MapReduce and MapReduce Job (MRJob) for developing large-scale deep learning models. We obtained about 5-30% reduction in the execution time of the deep auto-encoder model even on a single node Hadoop cluster. Furthermore, the complexity of code development is significantly reduced to create multi-layer deep learning models.

Performance evaluation of cloud service with hadoop for twitter data

<p>In the era of rapid growth of cloud computing, performance calculation of cloud service is an essential criterion to assure quality of service. Nevertheless, it is a perplexing task to effectively analyze the performance of cloud service due to the complexity of cloud resources and the diversity of Big Data applications. Hence, we propose to examine the performance of Big Data applications with Hadoop and thus to figure out the performance in cloud cluster. Hadoop is built based on MapReduce, one of the widely used programming models in Big Data. In this paper, the performance analysis of Hadoop MapReduce WordCount application for Twitter data is presented. A 4-node in-house Hadoop cluster was setup and experiment was carried out for analyzing the performance. Through this work, it was concluded that Hadoop is efficient for BigData applications with 3 or more nodes with replication factor 3. Also, it was observed that system time was relatively more compared to user time for BigData applications beyond 80GB. This experiment had also thrown certain pattern on actual data blocks used to process the WordCount application. </p>

An Energy Efficiency Optimization and Control Model for Hadoop Clusters

An Energy Efficiency Optimization and Control Model for Hadoop Clusters

Optimization of hadoop cluster foranalyzing large-scale sequence data inbioinformatics

Optimization of hadoop cluster foranalyzing large-scale sequence data inbioinformatics

Improved classification approach for use with large-scale scene images in the Hadoop cluster environment

Faced with massive amounts of image data, the performance of classification algorithms based on traditional platforms with single-node architecture drops dramatically. We propose a classification method based on hybrid optimization and combination technology in a cluster environment that is suitable for use with large-scale scene images. Support vector machine (SVM) algorithms are optimized by the artificial bee colony and particle swarm optimization algorithms to produce weak classifiers; then, a strong classifier is constructed by combining the outputs from the 15 weak classifiers using the AdaBoost algorithm. The MapReduce parallel programming model in the Hadoop platform is used to parallelize the algorithm, and a parallel AdaBoost hybrid optimization (PAH)-SVM algorithm is proposed. Finally, a model is constructed for automatic classification of the large-scale scene images. Multiple sets of comparative experiments show that the average classification accuracy of the proposed algorithm when applied to the scene understanding (Caltech-256 and Pascal VOC 2012) database exceeds 85.0%, and its training time is <10 min when 170,000 images are used. Considering the cost of hardware, the execution time and accuracy of this algorithm are superior to those of mainstream classification algorithms, such as P-SVM and CNN. In addition, the speed of the system based on the proposed algorithm increases linearly, and the constructed Hadoop cluster shows good extensibility. The proposed algorithm is suitable for automatic classification and prediction using large-scale scene images.

Research on MapReduce Task Scheduling Optimization

One of the big challenges in MapReduce is how to effectively allocate resources for the jobs submitted by users, and finish the job within the time specified by the user. The original scheduling strategy lacks the ability of job management and can not complete the job according to the time requirement of the user. In this paper, a task scheduler (ET-scheduler) is proposed to meet the needs of job time and resource optimization. It can not only meet the time needs of users, but also minimize the resource consumption and adjust the time allocation in the process of map and reduce. Experiments show that the algorithm not only completes the most jobs in a given time, but also minimizes the resource consumption in the Hadoop cluster.

An Empirical Performance Analysis on Hadoop via Optimizing the Network Heartbeat Period

To support a large-scale Hadoop cluster, Hadoop heartbeat messages are designed to deliver the significant messages, including task scheduling and completion messages, via piggybacking to reduce the number of messages received by the NameNode. Although Hadoop is designed and optimized for high-throughput computing via batch processing, the real-time processing of large amounts of data in Hadoop is increasingly important. This paper evaluates Hadoop’s performance and costs when the heartbeat period is controlled to support latency sensitive applications. Through an empirical study based on Hadoop 2.0 (YARN) [1] architecture, we improve Hadoop’s I/O performance as well as application performance by up to 13 percent compared to the default configuration. We offer a guideline that predicts the performance, costs and limitations of the total system by controlling the heartbeat period using simple equations. We show that Hive performance can be improved by tuning Hadoop’s heartbeat periods through extensive experiments.

Performance evaluation and resource optimization of cloud based parallel Hadoop clusters with an intelligent scheduler.

Data generated from real time information systems are always incremental in nature. Processing of such a huge incremental data in large scale requires a parallel processing system like Hadoop based cluster. Major challenge that arises in all cluster-based system is how efficiently the resources of the system can be used. The research carried out proposes a model architecture for Hadoop cluster with additional components integrated such as super node who manages the clusters computations and a mediation manager who does the performance monitoring and evaluation. Super node in the system is equipped with intelligent or adaptive scheduler that does the scheduling of the job with optimal resources. The scheduler is termed intelligent as it automatically decides which resource to be taken for which computation, with the help of a cross mapping of resource and job with a genetic algorithm which finds the best matching resource. The mediation node deploys ganglia a standard monitoring tool for Hadoop cluster to collect and record the performance parameters of the Hadoop cluster. The system over all does the scheduling of different jobs with optimal usage of resources thus achieving better efficiency compared to the native capacity scheduler in Hadoop. The system is deployed on top of OpenNebula Cloud environment for scalability.

Hadoop Security Challenges and Its Solution Using KNOX

&lt;p&gt;Big Data is a new technology and architecture. It can work on a very large volume of a variety of data with high-velocity, discovery, and/or analysis. Big Data is about the fast-growing sources of data such as web logics, Sensor networks, Social media, Internet text and documents, Internet pages, Search Index data, scientific research. Big data also formally introduces a complex range of analysis. Big data can evaluate mixed data (structured and unstructured) from multiple sources. As there are some security issues in big data which are no longer solved using the hashing techniques on large amount of data, this paper shows an idea of new approach of designing a Knox’ified Hadoop cluster.&lt;/p&gt;

DESIGN OF COMPREHENSIVE FRAMEWORK ON OPTIMIZATION METHODS IN DISTRIBUTED CLUSTERS

MapReduce is a popular, open source programming paradigm to handle big data which is an industry standard large scale data processing system used by many companies like Yahoo, Google, Facebook, etc. The YARN framework uses low resource fairness algorithms such as FIFO, Capacity, Fair, DRF scheduler, whereas these schedulers are not suitable for heterogeneous Hadoop clusters. Therefore, an Enhanced Combined Regression Ranking (eCRRYARN) algorithm was proposed to enhance resource fairness. The proposed algorithm uses linear regression model to estimate the expected resources to be availed by the tenants. The order ranking is given to the estimated resource and the resources shared as per the ranking provided. Hence, the Hierarchical Hadoop Cluster Resource Sharing (HHCRS) algorithm has been adopted for hierarchical distributed cluster aiming to design a cost effective cluster for organization which is spread across the globe.

Urban Planning and Smart City Decision Management Empowered by Real-Time Data Processing Using Big Data Analytics.

The Internet of Things (IoT), inspired by the tremendous growth of connected heterogeneous devices, has pioneered the notion of smart city. Various components, i.e., smart transportation, smart community, smart healthcare, smart grid, etc. which are integrated within smart city architecture aims to enrich the quality of life (QoL) of urban citizens. However, real-time processing requirements and exponential data growth withhold smart city realization. Therefore, herein we propose a Big Data analytics (BDA)-embedded experimental architecture for smart cities. Two major aspects are served by the BDA-embedded smart city. Firstly, it facilitates exploitation of urban Big Data (UBD) in planning, designing, and maintaining smart cities. Secondly, it occupies BDA to manage and process voluminous UBD to enhance the quality of urban services. Three tiers of the proposed architecture are liable for data aggregation, real-time data management, and service provisioning. Moreover, offline and online data processing tasks are further expedited by integrating data normalizing and data filtering techniques to the proposed work. By analyzing authenticated datasets, we obtained the threshold values required for urban planning and city operation management. Performance metrics in terms of online and offline data processing for the proposed dual-node Hadoop cluster is obtained using aforementioned authentic datasets. Throughput and processing time analysis performed with regard to existing works guarantee the performance superiority of the proposed work. Hence, we can claim the applicability and reliability of implementing proposed BDA-embedded smart city architecture in the real world.

Accommodation of curtailed wind power by electric water heaters based on a new hybrid prediction approach

Wind power curtailment is of great importance with the increase of large-scale wind power connected to the grid. A new concept of redundant wind power accommodated by dispatching electric water heaters (EWHs) is developed in the paper. Precise predictions of wind power and EWHs load power are the basis for this work. A hybrid multi-kernel prediction approach integrating an adaptive fruit fly optimization algorithm (AFOA) and multi-kernel relevance vector machine (MKRVM) is proposed to deal with the sample distribution of multi-source heterogeneous features uncovered by an energy entropy method, where AFOA is used to determine the kernel parameters in MKRVM adaptively and avoid the arbitrariness. For the large computation of the prediction approach, parallel computation based on the Hadoop cluster is used to accelerate the calculation. Then, an economic dispatching model for accommodating wind power is built taking into account the penalty of curtailed wind power and the operating cost of EWHs. The proposed scheme is implemented in an intelligent residential district. The results show that the optimization performance of the hybrid prediction approach is superior to those of four usual optimization algorithms in this case. Regular or orderly scheduling of EWHs enables accommodation of superfluous wind power and reduces dispatch cost.

Exploiting Apache Spark platform for CMS computing analytics

The CERN IT provides a set of Hadoop clusters featuring more than 5 PBytes of raw storage with different open-source, user-level tools available for analytical purposes. The CMS experiment started collecting a large set of computing meta-data, e.g. dataset, file access logs, since 2015. These records represent a valuable, yet scarcely investigated, set of information that needs to be cleaned, categorized and analyzed. CMS can use this information to discover useful patterns and enhance the overall efficiency of the distributed data, improve CPU and site utilization as well as tasks completion time. Here we present evaluation of Apache Spark platform for CMS needs. We discuss two main use-cases CMS analytics and ML studies where efficient process billions of records stored on HDFS plays an important role. We demonstrate that both Scala and Python (PySpark) APIs can be successfully used to execute extremely I/O intensive queries and provide valuable data insight from collected meta-data.