Hadoop Distributed File System Research Articles

Enhancing Hadoop distributed storage efficiency using multi-agent systems

<p>Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.</p>

Kafka‐Shield: Kafka Streams‐based distributed detection scheme for IoT traffic‐based DDoS attacks

AbstractWith the rapid proliferation of insecure Internet of Things (IoT) devices, the security of Internet‐based applications and networks has become a prominent concern. One of the most significant security threats encountered in IoT environments is a Distributed Denial of Service (DDoS) attack. This attack can severely disrupt critical services and prevent smart devices from functioning normally, leading to severe consequences for businesses and individuals. It aims to overwhelm victims' resources, websites, and other services by flooding them with massive attack packets, making them inaccessible to legitimate users. Researchers have developed multiple detection schemes to detect DDoS attacks. As technology advances and other facilitating factors have increased, it is a challenge to identify such powerful attacks in real‐time. In this paper, we propose a novel distributed detection scheme for IoT network traffic‐based DDoS attacks by deploying it in a Kafka Streams processing framework named Kafka‐Shield. The Kafka‐Shield comprises two stages: design and deployment. Firstly, the detection scheme is designed on the Hadoop cluster employing a highly scalable H2O.ai machine learning platform. Secondly, a portable, scalable, and distributed detection scheme is deployed on the Kafka Streams processing framework. To analyze the incoming traffic data and categorize it into nine target classes in real time. Additionally, Kafka‐Shield stores each network flow with significant input features and the predicted outcome in the Hadoop Distributed File System (HDFS). It enables the development of new models or updating current ones. To validate the effectiveness of the Kafka‐Shield, we performed critical analysis using various configured attack scenarios. The experimental results affirm Kafka‐Shield's remarkable efficiency in detecting DDoS attacks. It has a detection rate of over 99% and can process 0.928 million traces in nearly 3.027 s.

Survey on MapReduce Scheduler Algorithms in Hadoop Framework

Hadoop serves as a robust framework tailored for the storage and processing of vast data volumes across clusters. Its foundation lies in the Hadoop Distributed File System (HDFS) for data storage, complemented by the MapReduce paradigm for data processing. MapReduce, functioning as a parallel computing framework, operates adeptly on distributed clusters to manage large-scale data processing tasks. Within this framework, scheduling emerges as a pivotal aspect, influencing the overall system's performance and efficiency. The essence of scheduling in MapReduce lies in enhancing performance, minimizing response times, and optimizing resource allocation. This paper undertakes a systematic exploration of existing scheduling algorithms, offering a fresh classification and detailed examination of each category. Furthermore, the analysis delves into the core principles, objectives, as well as the strengths and weaknesses of these scheduling algorithms.

Anomaly detection in log-event sequences: A federated deep learning approach and open challenges

Anomaly Detection (AD) is an important area to reliably detect malicious behavior and attacks on computer systems. Log data is a rich source of information about systems and thus provides a suitable input for AD. With the sheer amount of log data available today, for years Machine Learning (ML) and more recently Deep Learning (DL) have been applied to create models for AD. Especially when processing complex log data, DL has shown some promising results in recent research to spot anomalies. It is necessary to group these log lines into log-event sequences, to detect anomalous patterns that span over multiple log lines. This work uses a centralized approach using a Long Short-Term Memory (LSTM) model for AD as its basis which is one of the most important approaches to represent long-range temporal dependencies in log-event sequences of arbitrary length. Therefore, we use past information to predict whether future events are normal or anomalous. For the LSTM model we adapt a state of the art open source implementation called LogDeep. For the evaluation, we use a Hadoop Distributed File System (HDFS) data set, which is well studied in current research. In this paper we show that without padding, which is a commonly used preprocessing step that strongly influences the AD process and artificially improves detection results and thus accuracy in lab testing, it is not possible to achieve the same high quality of results shown in literature. With the large quantity of log data, issues arise with the transfer of log data to a central entity where model computation can be done. Federated Learning (FL) tries to overcome this problem, by learning local models simultaneously on edge devices and overcome biases due to a lack of heterogeneity in training data through exchange of model parameters and finally arrive at a converging global model. Processing log data locally takes privacy and legal concerns into account, which could improve coordination and collaboration between researchers, cyber security companies, etc., in the future. Currently, there are only few scientific publications on log-based AD which use FL. Implementing FL gives the advantage of converging models even if the log data are heterogeneously distributed among participants as our results show. Furthermore, by varying individual LSTM model parameters, the results can be greatly improved. Further scientific research will be necessary to optimize FL approaches.

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments.

With the development of synchrotron radiation sources and high-frame-rate detectors, the amount of experimental data collected at synchrotron radiation beamlines has increased exponentially. As a result, data processing for synchrotron radiation experiments has entered the era of big data. It is becoming increasingly important for beamlines to have the capability to process large-scale data in parallel to keep up with the rapid growth of data. Currently, there is no set of data processing solutions based on the big data technology framework for beamlines. Apache Hadoop is a widely used distributed system architecture for solving the problem of massive data storage and computation. This paper presents a set of distributed data processing schemes for beamlines with experimental data using Hadoop. The Hadoop Distributed File System is utilized as the distributed file storage system, and Hadoop YARN serves as the resource scheduler for the distributed computing cluster. A distributed data processing pipeline that can carry out massively parallel computation is designed and developed using Hadoop Spark. The entire data processing platform adopts a distributed microservice architecture, which makes the system easy to expand, reduces module coupling and improves reliability.

KS-SDN-DDoS: A Kafka streams-based real-time DDoS attack classification approach for SDN environment

Software-Defined Networking (SDN) is a modern networking architecture that segregates control logic from data plane and supports a loosely coupled architecture. It provides flexibility in this advanced networking paradigm for any changes. Further, it controls the complete network in a centralized using controller(s). However, it comes with several security issues: Exhausting bandwidth and flow tables, Distributed Denial of Service (DDoS) attacks, etc. DDoS is a powerful attack for Internet-based applications and services, traditional and SDN paradigms. In the case of the SDN environment, attackers frequently target the central controller(s). This paper proposes a Kafka Streams-based real-time DDoS attacks classification approach for the SDN environment, named KS-SDN-DDoS. The KS-SDN-DDoS has been designed using highly scalable H2O ML techniques on the two-node Apache Hadoop Cluster (AHC). It consists of two modules: (i) Network Traffic Capture (NTCapture) and (ii) Attack Detection and Traffic Classification (ADTClassification). The NTCapture is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-1). It captures incoming network traffic, extracts and formulates attributes, and publishes significant network traffic attributes on the Kafka topic. The ADTClassification is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-2). It consumes network flows from the Kafka topic, classifies it based on the ten attributes, and publishes it to the decision Kafka topic. Further, it saves attributes with outcome to the Hadoop Distributed File System (HDFS). The KS-SDN-DDoS approach is designed and validated using the recent “DDoS Attack SDN dataset”. The result shows that the proposed system gives better classification accuracy (100%).

DATA FINDING, SHARING AND DUPLICATION REMOVAL IN THE CLOUD

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Design of storage benchmark kit framework for supporting the file storage retrieval

An open-source software framework called the storage benchmark kit (SBK) is used to store the system benchmarking performance framework. The SBK is designed to perform any storage client or device using any data type as a payload. SBK simultaneously helps number of readers as well as writes to the storage system of large amounts of data as well as allows end-to-end latency benchmarking for multiple writers and readers. The SBK uses standardized performance measures for comparing and evaluating various storage systems and their combinations. Distributed file systems, distributed database systems, single or local node databases, systems of object storage, platforms of distributed streaming and messaging, and systems of key-value storage are the storage solutions supported by SBK. The SBK supports various storage systems like XFS, Kafka streaming storage systems, and Hadoop distributed file system (HDFS) performance benchmarking. The experimental results show that a proposed method achieves execution time of 65.530 s, 40.826 s and 30.351 s for the 100k, 500k and 1000k files respectively which ensures better improvement than the existing methods such as simple data interface and distributed data protection system.

CLOUD BASED DUPLICATION REMOVAL SYSTEM

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Investigating optimal features in log files for anomaly detection using optimization approach

<p><span lang="EN-US">Logs have been frequently utilised in different software system administration activities. The number of logs has risen dramatically due to the vast scope and complexity of current software systems. A lot of research has been done on log-based anomaly identification using machine learning approach. In this paper, we proposed an optimization approach to select the optimal features from the logs. This will provide the higher classification accuracy on reduced log files. In order to predict the anomalies three phases are used: i) log representation ii) feature selection and iii) Performance evaluation. The efficacy of the proposed model is evaluated using benchmark datasets such as BlueGene/L (BGL), Thunderbird, spirit and hadoop distributed file system (HDFS) in terms of accuracy, converging ability, train and test accuracy, receiver operating characteristic (ROC) measures, precision, recall and F1-score. The results shows that the feature selection on log files outperforms in terms all the evaluation measures.</span></p>

A MapReduce-Based Approach for Fast Connected Components Detection from Large-Scale Networks.

Owing to increasing size of the real-world networks, their processing using classical techniques has become infeasible. The amount of storage and central processing unit time required for processing large networks is far beyond the capabilities of a high-end computing machine. Moreover, real-world network data are generally distributed in nature because they are collected and stored on distributed platforms. This has popularized the use of the MapReduce, a distributed data processing framework, for analyzing real-world network data. Existing MapReduce-based methods for connected components detection mainly struggle to minimize the number of MapReduce rounds and the amount of data generated and forwarded to the subsequent rounds. This article presents an efficient MapReduce-based approach for finding connected components, which does not forward the complete set of connected components to the subsequent rounds; instead, it writes them to the Hadoop Distributed File System as soon as they are found to reduce the amount of data forwarded to the subsequent rounds. It also presents an application of the proposed method in contact tracing. The proposed method is evaluated on several network data sets and compared with two state-of-the-art methods. The empirical results reveal that the proposed method performs significantly better and is scalable to find connected components in large-scale networks.

Securing cloud-enabled smart cities by detecting intrusion using spark-based stacking ensemble of machine learning algorithms

<abstract> <p>With the use of cloud computing, which provides the infrastructure necessary for the efficient delivery of smart city services to every citizen over the internet, intelligent systems may be readily integrated into smart cities and communicate with one another. Any smart system at home, in a car, or in the workplace can be remotely controlled and directed by the individual at any time. Continuous cloud service availability is becoming a critical subscriber requirement within smart cities. However, these cost-cutting measures and service improvements will make smart city cloud networks more vulnerable and at risk. The primary function of Intrusion Detection Systems (IDS) has gotten increasingly challenging due to the enormous proliferation of data created in cloud networks of smart cities. To alleviate these concerns, we provide a framework for automatic, reliable, and uninterrupted cloud availability of services for the network data security of intelligent connected devices. This framework enables IDS to defend against security threats and to provide services that meet the users' Quality of Service (QoS) expectations. This study's intrusion detection solution for cloud network data from smart cities employed Spark and Waikato Environment for Knowledge Analysis (WEKA). WEKA and Spark are linked and made scalable and distributed. The Hadoop Distributed File System (HDFS) storage advantages are combined with WEKA's Knowledge flow for processing cloud network data for smart cities. Utilizing HDFS components, WEKA's machine learning algorithms receive cloud network data from smart cities. This research utilizes the wrapper-based Feature Selection (FS) approach for IDS, employing both the Pigeon Inspired Optimizer (PIO) and the Particle Swarm Optimization (PSO). For classifying the cloud network traffic of smart cities, the tree-based Stacking Ensemble Method (SEM) of J48, Random Forest (RF), and eXtreme Gradient Boosting (XGBoost) are applied. Performance evaluations of our system were conducted using the UNSW-NB15 and NSL-KDD datasets. Our technique is superior to previous works in terms of sensitivity, specificity, precision, false positive rate (FPR), accuracy, F1 Score, and Matthews correlation coefficient (MCC).</p> </abstract>

Introducing an intelligent multi-level retrieval method for mineral resource potential evaluation result data

Abstract The geological data of the mineral resource potential evaluation results (MRPERs) are diverse and extremely large; efficiently retrieving data remains a challenging problem. In this work, a new way of using the Hadoop platform is proposed. The Hadoop distributed file system is used to store the massive data and construct the data storage model of geological and mineral resources. Using a distributed Hadoop database (HBase) that supports the fast query of a single record, it manages its metadata and retrieves the data of MRPERs quickly. At the same time, a multi-level index directory is designed to support the non-main key query on the HBase. This overcomes the shortcoming that the HBase only supports the simple index based on the main key and realizes the intelligent, efficient retrieval of MRPERs. The validity and feasibility of the proposed method are further verified by experiments using the MRPER data in the Institute of Mineral Resources, Chinese Academy of Geological Sciences.

Blockchain Enabled Hadoop Distributed File System Framework for Secure and Reliable Traceability

Hadoop Distributed File System (HDFS) is a distributed file system that allows large amounts of data to be stored and processed across multiple servers in a Hadoop cluster. HDFS also provides high throughput for data access. HDFS enables the management of vast amounts of data using commodity hardware. However, security vulnerabilities in HDFS can be manipulated for malicious purposes. This emphasizes the significance of establishing strong security measures to facilitate file sharing within Hadoop and implementing a reliable mechanism for verifying the legitimacy of shared files. The objective of this paper is to enhance the security of HDFS by utilizing a blockchain-based technique. The proposed model uses the Hyperledger Fabric platform at the enterprise level to leverage metadata of files, thereby establishing dependable security and traceability of data within HDFS. The analysis of results indicates that the proposed model incurs a slightly higher overhead compared to HDFS and requires more storage space. However, this is considered an acceptable trade-off for the improved security.

Cost-based Data Prefetching and Scheduling in Big Data Platforms over Tiered Storage Systems

The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN’s resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency.

A blockchain integration to support failures prediction from log files in multi-agent systems technology

Computer system failures can cause significant financial and reputational damage, making it critical to detect and prevent them before they occur. In this article, we propose a novel approach for predicting system failures based on log file analysis, called MABFailPred. Our approach leverages a Blockchain-based Multi-Agent architecture to analyze execution traces and detect early warning signals for system failure prediction during execution. Moreover, we introduce a new method for extracting log keys called HCLPars. The MABFailPred approach comprises four agents that work together to gather log keys, categorize data, develop a failure model, and make predictions. The use of blockchain ensures that agent and administrator predictions are synchronized, and agent collaboration is coordinated. We validate our approach by applying it to a real-world distributed system and accurately predicting problems before they occur. Our proposed approach represents a significant advancement in the field of system failure prediction, with the potential to enhance the reliability and security of computer systems. By employing a machine learning algorithm, our approach can predict anomalies in real-time and places a strong emphasis on the continuous monitoring of system behavior. The paper includes a validation of the approach using the Hadoop Distributed File System (HDFS) log dataset. According to our findings, the proposed approach achieves superior performance compared to other methods on the same dataset, with a precision of 0.998. This result establishes a new state-of-the-art performance level for the dataset.

SeQual-Stream: approaching stream processing to quality control of NGS datasets

BackgroundQuality control of DNA sequences is an important data preprocessing step in many genomic analyses. However, all existing parallel tools for this purpose are based on a batch processing model, needing to have the complete genetic dataset before processing can even begin. This limitation clearly hinders quality control performance in those scenarios where the dataset must be downloaded from a remote repository and/or copied to a distributed file system for its parallel processing.ResultsIn this paper we present SeQual-Stream, a streaming tool that allows performing multiple quality control operations on genomic datasets in a fast, distributed and scalable way. To do so, our approach relies on the Apache Spark framework and the Hadoop Distributed File System (HDFS) to fully exploit the stream paradigm and accelerate the preprocessing of large datasets as they are being downloaded and/or copied to HDFS. The experimental results have shown significant improvements in the execution times of SeQual-Stream when compared to a batch processing tool with similar quality control features, providing a maximum speedup of 2.7times when processing a dataset with more than 250 million DNA sequences, while also demonstrating good scalability features.ConclusionOur solution provides a more scalable and higher performance way to carry out quality control of large genomic datasets by taking advantage of stream processing features. The tool is distributed as free open-source software released under the GNU AGPLv3 license and is publicly available to download at https://github.com/UDC-GAC/SeQual-Stream.

EStore: A User-Friendly Encrypted Storage Scheme for Distributed File Systems.

In this paper, we propose a user-friendly encrypted storage scheme named EStore, which is based on the Hadoop distributed file system. Users can make use of cloud-based distributed file systems to collaborate with each other. However, most data are processed and stored in plaintext, which is out of the owner's control after it has been uploaded and shared. Meanwhile, simple encryption guarantees the confidentiality of uploaded data but reduces availability. Furthermore, it is difficult to deal with complex key management as there is the problem whereby a single key encrypts different files, thus increasing the risk of leakage. In order to solve the issues above, we put forward an encrypted storage model and a threat model, designed with corresponding system architecture to cope with these requirements. Further, we designed and implemented six sets of protocols to meet users' requirements for security and use. EStore manages users and their keys through registration and authentication, and we developed a searchable encryption module and encryption/decryption module to support ciphertext retrieval and secure data outsourcing, which will only minimally increase the calculation overhead of the client and storage redundancy. Users are invulnerable compared to the original file system. Finally, we conducted a security analysis of the protocols to demonstrate that EStore is feasible and secure.

Research on digital reform of music education resources in universities in the context of data governance

Abstract This paper proposes the use of the Hadoop-distributed file system HDFS as a storage system for large resources in music education. In order to improve the overall performance of HDFS for storing educational resources and to solve the deficiencies of the current system. This paper provides an overall analysis of the current state of research related to the placement algorithm and data storage strategy of HDFS, respectively. After combining the characteristics of educational resources themselves and establishing an educational resource value evaluation index system, an improved placement algorithm and a hybrid redundant storage strategy based on educational resource ratings are proposed, and the proposed algorithm or strategy is simulated and implemented and verified through source code analysis of HDFS-related algorithms. The results show that the average response time for uploading 10 songs is 7.734 s, and the average response time for image retrieval is 0.768 s. In the analysis of digital music education resources demand, teachers have the highest usage rate of music teaching theory (M=3.43), 38.45% of students’ demand is mainly for downloading music, and music lovers querying popular music has the highest percentage of 19.43%. The algorithm in this paper achieves the storage load balance of each node, which basically satisfies the demand for music education resources of teachers and students in universities.

A Heuristic Approach to Improve the Data Processing in Big Data using Enhanced Salp Swarm Algorithm (ESSA) and MK-means Algorithm

Improving data processing in big data is a delicate procedure in our current digital era due to the massive amounts of data created by humans and machines in daily life. Handling this data, creating a repository for storage, and retrieving photos from internet platforms is a difficult issue for businesses and industries. Currently, clusters have been constructed for many types of data, such as text, documents, audio, and video files, but the extraction time and accuracy during data processing remain stressful. Hadoop Distributed File System (HDFS) is a system that provides a large storage area in big data for managing large datasets, although the accuracy level is not as high as desired. Furthermore, query optimization was used to produce low latency and high throughput outcomes. To address these concerns, this study proposes a novel technique for query optimization termed the Enhanced Salp Swarm Algorithm (ESSA) in conjunction with the Modified K-Means Algorithm (MKM) for cluster construction. The process is separated into two stages: data collection and organization, followed by data extraction from the repository. Finally, numerous experiments with assessments were carried out, and the outcomes were compared. This strategy provides a more efficient method for enhancing data processing speed in a big data environment while maintaining an accuracy level of 98% while processing large amounts of data.