Hadoop Cluster Research Articles

Novel data‐placement scheme for improving the data locality of Hadoop in heterogeneous environments

SummaryTo address the challenging needs of high‐performance big data processing, parallel‐distributed frameworks such as Hadoop are being utilized extensively. However, in heterogeneous environments, the performance of Hadoop clusters is below par. This is primarily because the blocks of the clusters are allocated equally to all nodes without regard to differences in the capability of individual nodes. This results in reduced data locality. Thus, a new data‐placement scheme that enhances data locality is required for Hadoop in heterogeneous environments. This article proposes a new data placement scheme that preserves the same degree of data locality in heterogeneous environments as that of the standard Hadoop, with only a small amount of replicated data. In the proposed scheme, only those blocks with the highest probability of being accessed remotely are selected and replicated. The results of experiments conducted indicate that the proposed scheme incurs only a 20% disk space overhead and has virtually the same data locality ratio as the standard Hadoop, which has a replication factor of three and 200% disk space overhead.

A Paucity of Code Status Documentation Despite Increasing Complex Chronic Disease in Pediatrics

Objectives: An increasing number of children are living with complex chronic diseases (CCDs) due to medical advances. Despite a need for code status discussions (CSDs), there is great variability in the frequency and documentation of such conversations. The objective was to identify gaps in the documentation of CSDs within the electronic health record (EHR), focusing on patients with CCDs. Methods: This was a retrospective review of all patients admitted from the emergency department of a tertiary care children's hospital in 2016. An EHR query using the Apache Hadoop cluster and manual review identified documentation of CSDs, including (1) code status orders, (2) advance directives, and (3) CSDs in provider notes. Patient complexity was stratified using the Pediatric Medical Complexity Algorithm 3.0. Comparative analysis was performed using chi-square, Kruskal-Wallis tests and multivariable logistic regression. Results: There were 12,648 unique patients of whom 4157 (32.9%) had CCD. Only 209 (1.7%) patients had a code status documented, of whom 200 (95.7%) had CCD. Of 528 (4.2%) patients ≥18 years of age, 428 (81.1%) had CCD and only 65 (12.3%) had CSDs. Palliative care consultation increased odds of CSDs (OR: 21.4, 95% CI: 13.8-33.2, p < 0.0001), whereas African American race decreased odds of CSDs (OR: 0.42, 95% CI: 0.27-0.64, p < 0.0001). Conclusions: Among admitted pediatric patients, most do not have documentation of CSDs, including those with CCD and patients ≥18 years of age. Improvements in both frequency and consistency of CSD documentation are needed to inform the family-centered care of patients living with CCDs.

A unit-based, cost-efficient scheduler for heterogeneous Hadoop systems

A significant amount of research in the field of job scheduling is carried out in Hadoop. However, there is still need for research to overcome some challenges regarding scheduling jobs in Hadoop clusters. There are various factors affecting the performance of scheduling policies like data volume (storage), data source format (different data), speed (data rate), security and privacy, cost, connection and data sharing. To reach a better utilization of resources and managing big data, scheduling policies have been designed. In this paper, an algorithm has been presented that can run on heterogeneous Hadoop clusters and runs job in parallel. This algorithm first distributes data based on the performance of the nodes and then schedules the jobs according to their cost of execution and decreases the cost of executing the jobs. The presented algorithm offers better performance in terms of execution time, cost and locality compared to FIFO and Fair schedulers.

Scalable Mining of Contextual Outliers Using Relevant Subspace

In this paper, we propose a scalable mining algorithm to discover contextual outliers using relevant subspaces. We develop the mining algorithm using the MapReduce programming model running on a Hadoop cluster. Relevant subspaces, which effectively capture the local distribution of various datasets, are quantified using local sparseness of attribute dimensions. We design a novel way of calculating local outlier factors in a relevant subspace with the probability density of local datasets; this new approach can effectively reflect the outlier degree of a data object that does not satisfy the distribution of the local dataset in the relevant subspace. Attribute dimensions of a relevant subspace, and local outlier factors are expressed as vital contextual information, which improves the interpretability of outliers. Importantly, the selection of ${N}$ data objects with the largest local outlier factor value is categorized as contextual outliers in our solution. To this end, our scalable mining algorithm, which incorporates the locality sensitive hashing distributed strategy, is implemented on a Hadoop cluster. The experimental results validate the effectiveness, interpretability, scalability, and extensibility of the algorithm using both synthetic data and stellar spectral data as experimental datasets.

Job failure prediction in Hadoop based on log file analysis

Hadoop is a popular framework based on MapReduce programming model to allow for distributed processing of large datasets across clusters with various number of computer nodes. Just like any dynamic computational environment, Hadoop has some problems and one of which is unsuccessful execution of MapReduce jobs. Job failures can cause significant resource wasting, performance deterioration, and user dissatisfaction. Therefore, a proactive and predictive management approach could be very useful in Hadoop systems. In this paper, we try to predict the futurity of MapReduce jobs in OpenCloud Hadoop cluster by using its log files. OpenCloud is a research cluster managed by CMU’s Parallel Data Lab which uses Hadoop to process big data. We first tried to study the log files and analyze the relationship between the jobs, resources, and workload characteristics and the failures in order to discover the effective features for the prediction process. After recognizing the job failure patterns, some popular machine learning algorithms are deployed to predict the success/failure status of the jobs before they start to execute. Eventually, we compared the learning methods and showed that the C5.0 algorithm had the best results with an accuracy of 91.37%, a recall of 74.43%, and a precision of 80.31%.

From Big Data to business analytics: The case study of churn prediction

The success of companies hugely depends on how well they can analyze the available data and extract meaningful knowledge. The Extract-Transform-Load (ETL) process is instrumental in accomplishing these goals, but requires significant effort, especially for Big Data. Previous works have failed to formalize, integrate, and evaluate the ETL process for Big Data problems in a scalable and cost-effective way. In this paper, we propose a cloud-based ETL framework for data fusion and aggregation from a variety of sources. Next, we define three scenarios regarding data aggregation during ETL: (i) ETL with no aggregation; (ii) aggregation based on predefined columns or time intervals; and (iii) aggregation within single user sessions spanning over arbitrary time intervals. The third scenario is very valuable in the context of feature engineering, making it possible to define features as “the time since the last occurrence of event X”. The scalability was evaluated on Amazon AWS Hadoop clusters by processing user logs collected with Kinesis streams with datasets ranging from 30 GB to 2.6 TB. The business value of the architecture was demonstrated with applications in churn prediction, service-outage prediction, fraud detection, and more generally — decision support and recommendation systems. In the churn prediction case, we showed that over 98% of churners could be detected, while identifying the individual reason. This allowed support and sales teams to perform targeted retention measures.

Research on campus network cloud storage open platform based on cloud computing and big data technology

Based on the analysis of campus cloud storage requirements, the paper plans a cloud storage structure model to achieve the goal of breaking the bottleneck of traditional campus network resource utilization. The model established in the thesis fully considers the problems of cluster data security an d big data analysis and utilization, and proposes an optimization scheme for cloud storage open platform system design. Plan the HDFS + HBase distributed storage deployment in the Hadoop cluster, elaborate the process of behavior analysis, and find a way to optimize and improve the cloud storage in colleges and universities. HBase technology implements operations on HDFS distributed storage systems in Hadoop clusters, effectively improving the efficiency of resource access and resource analysis. The cloud platform storage platform designed by the thesis solves the problem of integrating the superior resources of colleges and universities.

Massive Power Information Processing Scheme Based on MongoDB

In order to solve the problem that the power consumption data presents a trend of sea quantification, and the huge power consumption data brings tremendous pressure to the system, especially slows down the speed of data query and gives users poor experience in using it, this paper proposes to use one of the non-relational databases (NoSQL) to store power consumption data instead of the traditional relational databases. To solve the problem that the collected electricity data can not meet the demand of page display, Hadoop MapReduce is proposed to preprocess the electricity data. Distributed architecture is adopted for storage and processing. MongoDB cluster and Hadoop cluster are overlapped and deployed. Combined with MongoDB’s powerful storage capacity and Hadoop MapReduce’s analytical and computational capability, a set of high-availability and high-performance data storage and pre-processing scheme for power-consuming universities is constructed.

K‐PSO: An improved PSO‐based container scheduling algorithm for big data applications

SummaryIn recent years, Docker container technology is being applied in the field of cloud computing at an explosive speed. The scheduling of Docker container resources has gradually become a research hotspot. Existing big data computing and storage platforms apply with traditional virtual machine technology, which often results in low resource utilization, a long time for flexible scaling and expanding clusters. In this paper, we propose an improved container scheduling algorithm for big data applications named Kubernetes‐based particle swarm optimization(K‐PSO). Experimental results show that the proposed K‐PSO algorithm converges faster than the basic PSO algorithm, and the running time of the algorithm is cut in about half. The K‐PSO container scheduling algorithm and algorithm experiment for big data applications are implemented in the Kubernetes container cloud system. Our experimental results show that the node resource utilization rate of the improved scheduling strategy based on K‐PSO algorithm is about 20% higher than that of the Kube‐scheduler default strategy, balanced QoS priority strategy, ESS strategy, and PSO strategy, while the average I/O performance and average computing performance of Hadoop cluster are not degraded.

Fine-grained data-locality aware MapReduce job scheduler in a virtualized environment

Big data overwhelmed industries and research sectors. Reliable decision making is always a challenging task, which requires cost-effective big data processing tools. Hadoop MapReduce is being used to store and process huge volume of data in a distributed environment. However, due to huge capital investment and lack of expertise to set up an on-premise Hadoop cluster, big data users seek cloud-based MapReduce service over the Internet. Mostly, MapReduce on a cluster of virtual machines is offered as a service for a pay-per-use basis. Virtual machines in MapReduce virtual cluster reside in different physical machines and co-locate with other non-MapReduce VMs. This causes to share IO resources such as disk and network bandwidth, leading to congestion as most of the MapReduce jobs are disk and network intensive. Especially, the shuffle phase in MapReduce execution sequence consumes huge network bandwidth in a multi-tenant environment. This results in increased job latency and bandwidth consumption cost. Therefore, it is essential to minimize the amount of intermediate data in the shuffle phase rather than supplying more network bandwidth that results in increased service cost. Considering this objective, we extended multi-level per node combiner for a batch of MapReduce jobs to improve makespan. We observed that makespan is improved up to 32.4 $$\%$$ by minimizing the number of intermediate data in shuffle phase when compared to classical schedulers with default combiners.

Processing streams in a monitoring cloud cluster

The creation of monitoring clusters based on cloud computing technologies is a promising direction for the development of systems for continuous monitoring of objects for various purposes in the web space. Hadoop web-programming environment is the technological basis for the development of algorithmic and software solutions for the synthesis of monitoring clusters, including information security and information counteraction systems. The International Telecommunication Union’ (ITU) recommendations Y. 3510 present the requirements for cloud infrastructure that require monitoring the performance of deployed applications based on the collection of real-world statistics. Often, computing resources of monitoring clusters of cloud data centers are allocated for continuous parallel processing of high-speed streaming data, which imposes new requirements to monitoring technologies, necessitating the creation and research of new models of parallel computing. The need to use service monitoring plays an important role in the cloud computing industry, especially for SLA/QoS assessment, as the application or service may experience problems even if the virtual machines on which the work is taking place appear to be operational. This requires to study the methodological possibilities of organization to study of parallel processing high-speed streaming services with the processing of huge amounts of bit data, and, simultaneously, to estimate the necessary computational resource. In the conditions of high dynamics of changes in the bit rate of information generation from the source, a model of the bit rate of Discretized Stream (DStream) formation is proposed, which has a common application. Based on the poly-burst nature of the bit rate model, a model of group content traffic of any sources of different services processed in the cloud cluster was created. The obtained results made it possible to develop mathematical models of parallel DStreams from sources processed in a cloud cluster via Hadoop technology using the micro-batch architecture of the Spark Streaming module. These models take into account the flow of requests for maintenance from sources of different services, on the one hand, and, on the other hand, the needs of services in bit rate, taking into account the multichannel traffic of sources of various services. At the same time, analytical relations are obtained to calculate the required performance of the Hadoop cluster at a given value of the probability of batch loss.

A two steps method of resources utilization predication for large Hadoop data center

SummaryWith the increase of data processing and Hadoop data center construction requirements, the performance of Hadoop data center is limited by inappropriate resources utilization. This paper introduces a new method to predict utilization for large‐scale Hadoop clusters. The new method adopts a two steps model, which includes Hadoop applications' performance simulation and resources utilization prediction. For performance simulation, a new simulator, which integrates baseline test and multilayered network model, is introduced and implemented. A resources utilization predictor is proposed in the second step. By analyzing the pattern of resources utilization, a single task model is proposed. A parallel‐batch‐task‐based (PBT) model, which represents the behavior of real Hadoop applications by integrating the single task model, is introduced. Two test scenarios are configured to verify the performance of our method. For the data center scenario, Terasort, Wordcount, and Hive are selected as benchmarks. In the virtual machines scenario, Terasort is used as benchmark. The experiments show that the error comparing between the simulator results and experimental environment results in most cases is less than 10%. The results confirm that we can locate the resource bottleneck for Hadoop clusters, meanwhile we can agilely configure clusters for applications with massive data.

Secure and privacy-preserving multi-keyword ranked information retrieval from encrypted big data

Cloud deployment raises some security challenges to the confidentiality of data, and the privacy of users. These challenges, along with the pressing demand for adopting big data technologies, together call for the development of stronger encryption algorithms. But encrypting the data makes it difficult to retrieve the most matching documents with respect to the query keywords. Therefore, the authors propose a solution for the ranked encrypted information retrieval, using the modified homomorphic encryption scheme (MHE) still preserving user's privacy. The scheme efficiently utilises the processing power of the cloud server to compute the similarity scores, leaving the decryption and ranking to the client side, thus ensuring the security of the data. Vector space model and term frequency-inverse document frequency (TF-IDF) concepts are used for similarity matching. The execution is then accelerated using a Hadoop cluster and is found to be accurate, efficient, scalable and practical for real world applications.

Teaching Hadoop Using Role Play Games

ABSTRACTHadoop is a well‐known big data system and a subject covered in many big data courses. This article describes two role play games for teaching the two fundamental components in the Hadoop framework, MapReduce and Hadoop Distributed File System (HDFS). In the games, students form teams and play different roles as a part of a Hadoop cluster. The games are designed to let students collaborate with peers in the same way as MapReduce and HDFS components collaborate to perform computing jobs in a Hadoop cluster. Utilizing a computer communication channel, the games are designed to let students quickly and effectively understand typical MapReduce and HDFS operations. Survey results from students in two big data classes show that the games effectively improve learning outcome in understanding MapReduce and HDFS workflow and the Hadoop framework in general. Students appear more engaged in class activities and communicate with peers more often.

Secure and privacy-preserving multi-keyword ranked information retrieval from encrypted big data

Cloud deployment raises some security challenges to the confidentiality of data, and the privacy of users. These challenges, along with the pressing demand for adopting big data technologies, together call for the development of stronger encryption algorithms. But encrypting the data makes it difficult to retrieve the most matching documents with respect to the query keywords. Therefore, the authors propose a solution for the ranked encrypted information retrieval, using the modified homomorphic encryption scheme (MHE) still preserving user's privacy. The scheme efficiently utilises the processing power of the cloud server to compute the similarity scores, leaving the decryption and ranking to the client side, thus ensuring the security of the data. Vector space model and term frequency-inverse document frequency (TF-IDF) concepts are used for similarity matching. The execution is then accelerated using a Hadoop cluster and is found to be accurate, efficient, scalable and practical for real world applications.

Fault prediction for distributed computing Hadoop clusters using real-time higher order differential inputs to SVM: Zedacross

Hadoop distributed computing clusters are used worldwide for high-performance computations. Often various hardware and software faults occur, leading to both data and computation time losses. This paper proposes the usage of a fault prediction software called 'Zedacross' which uses machine learning principles combined with cluster monitoring tools. Firstly, the paper suggests a model that uses the resource usage statistics of a normally functioning Hadoop cluster to create a machine learning model that can then be used to predict and detect faults in real time. Secondly, the paper explains the novel idea of using higher order differentials as inputs to SVM for highly accurate fault predictions. Predictions of system faults by observing system resource usage statistics in real-time with minimum delay will play a vital role in deciding the need for job rescheduling tasks or even dynamic up-scaling of the cluster. To demonstrate the effectiveness of the design a Java utility was built to perform cluster fault monitoring. The results obtained after running the system on various test cases demonstrate that the proposed method is accurate and effective.

Fault prediction for distributed computing Hadoop clusters using real-time higher order differential inputs to SVM: Zedacross

Hadoop distributed computing clusters are used worldwide for high-performance computations. Often various hardware and software faults occur, leading to both data and computation time losses. This paper proposes the usage of a fault prediction software called 'Zedacross' which uses machine learning principles combined with cluster monitoring tools. Firstly, the paper suggests a model that uses the resource usage statistics of a normally functioning Hadoop cluster to create a machine learning model that can then be used to predict and detect faults in real time. Secondly, the paper explains the novel idea of using higher order differentials as inputs to SVM for highly accurate fault predictions. Predictions of system faults by observing system resource usage statistics in real-time with minimum delay will play a vital role in deciding the need for job rescheduling tasks or even dynamic up-scaling of the cluster. To demonstrate the effectiveness of the design a Java utility was built to perform cluster fault monitoring. The results obtained after running the system on various test cases demonstrate that the proposed method is accurate and effective.

Performance tuning analysis of spatial operations on Spatial Hadoop cluster with SSD

Solid State Drives have shown promising results in improving the performance of MapReduce jobs. Previous work in the usage of SSD consisted of evaluation of its incorporation in Hadoop using benchmarking tests, such as WordCount, TeraGen, TeraSort and so on. It was empirically observed that the type of Map Reduce implementation, specifically the nature of the processing performed during the shuffle phase, has a major role in the overall outcome. This finding opens a plethora of possibilities of exploring the impact of storage media on spatial data processing in Big Data frameworks. Spatial Big data frameworks, such as Spatial Hadoop, have been successfully used for several spatial operations, but the effect of SSD utilization in these operations is yet to be evaluated. This paper evaluates the feasibility of adopting SSD in a Spatial Hadoop cluster. The contributions presented are 1) Characterization of Spatial Hadoop jobs that would benefit from the usage of SSD. 2) Identification of the optimal mix of Solid State Drives and Hard Disk Drives for HDFS for each type of job. 3) Evaluation of cluster configuration parameters that influence the speed-up. The cluster exhibited a speed-up of up to 15% for I/O intensive spatial operations, while other compute-intensive operations could not harness a similar gain.

Hardware and software data processing system for research and scientific purposes based on Raspberry Pi 3 microcomputer

In the past ten years, rapid progress has been observed in science and technology through the development of smart mobile devices, workstations, supercomputers, smart gadgets and network servers. Increase in the number of Internet users and a multiple increase in the speed of the Internet led to the generation of a huge amount of data, which is now commonly called «big data». Given this scenario, storing and processing data on local servers or personal computers can cause a number of problems that can be solved using distributed computing, distributed data storage and distributed data transfer. There are currently several cloud service providers to solve these problems, like Amazon Web Services, Microsoft Azure, Cloudera and etc. Approaches for distributed computing are supported using powerful data processing centers (DPCs). However, traditional DPCs require expensive equipment, a large amount of energy to run and operate the system, a powerful cooling system and occupy a large area. In addition, to maintain such a system, its constant use is necessary, because its stand-by is economically disadvantageous. The article is aimed at the possibility of using a Raspberry Pi and Hadoop cluster for distributed storage and processing of «big data». Such a trip provides low power consumption, the use of limited physical space, high-speed solution to the problems of processing data. Hadoop provides the necessary modules for distributed processing of big data by deploying Map-Reduce software approaches. Data is stored using the Hadoop Distributed File System (HDFS), which provides more flexibility and greater scalability than a single computer. The proposed hardware and software data processing system based on Raspberry Pi 3 microcomputer can be used for research and scientific purposes at universities and scientific centers. Considered distributed system shows economically efficiency in comparison to traditional DPCs. The results of pilot project of Raspberry Pi cluster application are presented. A distinctive feature of this work is the use of distributed computing systems on single-board microcomputers for academic purposes for research and educational tasks of students with minimal cost and ease of creating and using the system.

Resilient intrusion detection system for cloud containers

The lightweight virtualisation and isolated execution offered by Linux containers qualify it to be the dominant virtualisation platform for cloud-based applications. The fact that Linux containers run on the same host while sharing the same kernel opens the door for new attacks. However, limited research has been conducted in the area of securing cloud containers. This paper presents a resilient intrusion detection and resolution system for cloud-based containers. The system relies on two main pillars, a real-time smart behaviour monitoring mechanism to detect maliciously behaving containers, and a moving-target defence approach that applies runtime container migration to quarantine such containers and to minimise attack dispersion. To avoid zero-day targeted attacks, the system also induces random live migrations between running containers to obfuscate its execution behaviour. Such obfuscation makes it harder for attackers to execute their targeted attacks. The system was tested by a big-data application using a container-based Apache Hadoop cluster to demonstrate the system's ability to automatically deploy, monitor, detect, and respond to maliciously behaving applications by live migration or by rolling back the container to a safe state. Results showed that the proposed system efficiently ensure safe and secure container operation.