Apache Hadoop Distributed File System Research Articles

PRBP: A prioritized replica balancing policy for HDFS balancer

AbstractData replication is the main fault tolerance mechanism implemented by the Apache Hadoop Distributed File System (HDFS). The placement of the data across the cluster directly affects replica balancing and data locality. The HDFS Balancer is the native solution to rebalance the data distribution by moving the blocks from over‐utilized to under‐utilized nodes. Nevertheless, its current balancing policy does not address the characteristics and specific needs of the applications during data rearrangement. In this work, we present the PRBP, a customized replica balancing policy for the HDFS Balancer. The PRBP is based on a system of priorities, which can be adapted and configured according to different demands of use, either these are related to heterogeneous environments or focused on improving data reliability and availability. The priorities define whether system metrics or aspects of the cluster topology should be considered during the execution of the HDFS Balancer, thus making the process of replica balancing in HDFS more flexible. Based on the priority system, we determine association rules that allow the use of multiple priorities simultaneously. Along with these rules, we present guidelines for using the PRBP as a specialized solution in scenarios that can benefit from reactive replica balancing. In addition, we conducted a practical experimentation to highlight the behavior and the applicability of the guidelines of the PRBP to prioritize replica rearrangement in the file system.

Big Data technologies to process spatial and attribute data when designing and operating mine-engineering systems

ABSTRACTA myriad of spatial and attribute data are in use when designing and operating mine-engineering systems. These data are hosted on various servers of a company and are diverse in structure of data and units of measurement and have different subject areas related to each other in a complex way. There is no conformity between conventional systems for managing and storing data and tools for analysing incoming data. Big Data paradigm that implies methods for processing distributed data is utilised in ‘big data’ processing. A MapReduce method involves two procedures: map procedure that applies a proper function to each element of the list and reduce procedure, integrating the map procedure results. Such conventional database management system methods as integration and indexation, graph search and other methods are used to cluster big data. These methods should be used within MapReduce. Employing standard equipment and means to control a distributed Hadoop and Apache Hadoop Distributed File System (HDFS) file system, data storages in petabytes can be implemented. To make a more in-depth analysis, Data Mining methods are utilised along with network analysis, predictive analytics, etc. The articlecovers various methods applied to collect and analyse ‘big data’ to do a feasibility study and design mine and engineering systems.

Using the Big Data in the human resources management systems

Measures like setting a destination, evaluation of performances, planning of the labour force about business, personnel recruitment, charging of personnel, informations about personnel, analytical of labour force and reporting, etc., are considered as vital problems of the human resources management. In this study, these criteria in a variety of data were collected with a large repository Apache Hadoop Distributed File System file system owned. Data entry and analysis were used Apache Pig and Java programming languages. The aim of the study is to help ‘the owners of business’ evaluate the abundance of data and to get rid of the management complexity via an application which is the biggest problem of management system in human resources. Keywords: Big Data, management of human resources, Hadoop, Pig, map-reduce.

Cloud Forensics : Isolating Cloud Instance

Abstract—Cloud computing has been the trending model for storing, accessing and modifying the data over the Internet in the recent years. Rising use of the cloud has generated a new concept related to the cloud which is cloud forensics. Cloud forensics can be defined as investigating for evidence over the cloud, so it can be viewed as a combination of both cloud computing and digital forensics. Many issues of applying forensics in the cloud have been addressed. Isolating the location of the incident has become an essential part of forensic process. This is done to ensure that evidence will not be modified or changed. Isolating an instant in the cloud computing has become even more challenging, due to the nature of the cloud environment. In the cloud, the same storage or virtual machine have been used by many users. Hence, the evidence is most likely will be overwritten and lost. The proposed solution in this paper is to isolate a cloud instance. This can be achieved by marking the instant that reside in the servers as "Under Investigation". To do so, cloud file system must be studied. One of the well-known file systems used in the cloud is Apache Hadoop Distributed File System (HDFS). Thus, in this paper the methodology used for isolating a cloud instance would be based on the HDFS architecture.
 Keywords: cloud computing; digital forensics; cloud forensics

Towards a trusted HDFS storage platform: Mitigating threats to Hadoop infrastructures using hardware-accelerated encryption with TPM-rooted key protection

As a follow-on to the authors' previous work, this paper further expands on the concept of creating a trusted Apache Hadoop Distributed File System (HDFS). We discuss our motivation and evaluate a threat model for HDFS, and address a set of common security concerns within HDFS through infrastructure and software involving data-at-rest encryption and integrity validation. To accomplish these goals, we make use of technology from the Trusted Computing Group, such as the pervasively available Trusted Platform Module. In addition, we discuss our design considerations in building an encryption framework for Hadoop in a trustworthy manner, and describe the results of our experiments creating an encryption scheme for Hadoop utilizing hardware key protections and AES-NI for encryption acceleration. As part of this design we evaluate the recently implemented crypto framework for Hadoop and independently test the performance claims of AES-NI regarding mitigating performance overhead.

Big Data Analysis with Dataset Scaling in Yet Another Resource Negotiator (YARN)

data is exceedingly large day by day. In some organizations, there is a need to analyze and process the gigantic data. This is a big data problem often faced by these organizations. It is not possible for single machine to handle that data. So we have used Apache Hadoop Distributed File System (HDFS) for storage and analysis. This paper shows experimental work done on the MapReduce Application on Health sector dataset. The result shows the behavior of the MapReduce application framework to map and reduce the big volume of the data. The main problem is to check the behavior of the MapReduce applications by increasing the size of dataset. Our analysis lies in understanding the Apache MapReduce application performance. We expect that execution time increases linearly with the dataset size but our analysis shows sometimes the execution time varies non- linearly with the increase in the dataset size. The experimental result shows that with scaling the datasets execution time distinguishes. KeywordsData, Hadoop, MapReduce, YARN, Single Node, Multi