Hadoop Distributed File System Research Articles

IoT Forensic Cyber Activities Detection and Prevention with Automated Machine Learning Model

The Internet of Things (IoT) has been deployed in a vast range of applications with exponential increases in data size and complexity. Existing forensic techniques are not effective for the accuracy and detection rate of security issues in IoT forensics. Cyber forensic comprises huge volume constraints that are processing huge volumes of data in the Information and Communication Technology (ICT) comprised of IoT devices and platforms. Trust blockchain is effective technology those are utilized to assess the tamper-proof records in all transaction in the IoT environment. With the implementation of trust blockchain the record and transaction are processed with a distributed ledger that is managed by the network nodes. The challenge associated with the trust blockchain in IoT forensics is cost and security. To achieve significant cost-effectiveness organizations, need to evaluate the risks and benefits associated with IoT forensics in the trust blockchain technology. In this paper, developed a Block Chain Enabled Cyber-Physical system with distributed storage. The developed Blockchain model is termed as Integrated Hadoop Blockchain Forensic Machin Learning (IHBF-ML). The IHBF-ML model uses the Hadoop Distributed File System (HDFS) with cyberspace to improve security. Within the IHBF-ML model, IoT data communication is established with the smart contract. The smart contract-based blockchain process uses the Machine Learning model integrated with Cat Boost classification model for anomaly detection. Cost in IoT forensic is minimized with the parallel processing of the data through MapReduce Framework for the traffic translation, extraction, and analysis of the dynamic feature traffic from the IoT environment. The experimental analysis stated that constructed IHBF-ML model reduces the cost by ~25% than the other conventional blockchain Ethereum and EOS.

RADAR-Pipeline: Scalable Feature Generation for Mobile Health Data

Introduction & BackgroundRADAR-Pipeline is an open-source Python framework designed to simplify and enhance mobile health data analysis. It has been designed to efficiently read and process the large amount of data generated through the RADAR-Base platform. RADAR-base is a scalable, real-time streaming and analytics open-source platform to facilitate research access and customisation requirements. Studies using the Radar-base platform have collected fine-grained longitudinal data from wearables and phones. The data can potentially create multitudes of digital biomarkers, which can be used to inform us greatly about the disease condition. Due to the sheer size of the data, it can be difficult for researchers to read and process those data -- a common task is identifying useful features and common data processing/analysis steps previously used by the community. Up to now, these have been hand-crafted by individual data scientists, often lacking the capability to be easily reused by the community without author-specific knowledge. Furthermore, generating variables based on already established research on a larger scale can be challenging and could hinder replication. Hence, we have designed RADAR-Pipeline to help researchers overcome these challenges. It empowers them to create and share their data analysis and visualisation pipelines, fostering collaboration and knowledge sharing within the research community. Objectives & ApproachThe primary objective of RADAR-Pipeline is to offer researchers a user-friendly and powerful platform to develop and share their research. Researchers can build reusable analysis and visualisation pipelines to ensure consistent and reliable results. It simplifies big data analysis by leveraging Apache Spark to handle large and complex mobile health datasets efficiently. Researchers can also save time and effort by reusing and extending existing pipelines built by others. Finally, the RADAR-Pipeline promotes collaboration and recognition by allowing researchers to share their work through the RADAR-base Analytics Catalogue, making their pipelines citable and accessible to the wider research community. Whilst Radar-pipeline has been designed to read data from Radar-base, it can also be used to read data from any dataset which uses Hadoop Distributed File System (HDFS) file system namespace. Relevance to Digital FootprintsMobile health data is rich and valuable for understanding human behaviour and health. RADAR-Pipeline addresses the challenges associated with analysing large and complex mobile health datasets, enabling researchers to extract valuable insights that can be used to (1) Improve public health: By enabling efficient analysis of large-scale mobile health data, RADAR-Pipeline can contribute to research efforts aimed at improving population health outcomes and developing effective interventions; (2) Personalised healthcare: By facilitating the extraction of individual-level features from mobile health data, RADAR-Pipeline can seamlessly be integrated with Kafka data streams and machine learning pipelines to process the data in real-time, which can then be utilised to create more effective and targeted real-time interventions. (3) Promote reproducible research: The framework's emphasis on transparency and reproducibility in research aligns with the conference's focus on the responsible use of digital mobile health data. Conclusions & ImplicationsRADAR-Pipeline is a valuable tool for researchers, offering them the means to harness the potential of mobile health data. By adopting this framework, researchers can achieve efficient and scalable data analysis, thereby streamlining the extracting insights from digital footprints. This efficiency enables researchers to delve deeper into the data and uncover valuable patterns and trends. Furthermore, RADAR-Pipeline promotes collaboration and knowledge sharing within the research community. By providing a standardised framework for data analysis, RADAR-Pipeline facilitates collaboration among researchers, leading to the sharing of best practices and the dissemination of knowledge.

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

The volume of data is growing at an astonishingly high speed. Traditional techniques for storing and processing data, such as relational and centralized databases, have become inefficient and time-consuming. Linked data and the Semantic Web make internet data machine-readable. Because of the increasing volume of linked data and Semantic Web data, storing and working with them using traditional approaches is not enough, and this causes limited hardware resources. To solve this problem, storing datasets using distributed and clustered methods is essential. Hadoop can store datasets because it can use many hard disks for distributed data clustering; Apache Spark can be used for parallel data processing more efficiently than Hadoop MapReduce because Spark uses memory instead of the hard disk. Semantic Web data has been stored and processed in this paper using Apache Spark GraphX and the Hadoop Distributed File System (HDFS). Spark's in-memory processing and distributed computing enable efficient data analysis of massive datasets stored in HDFS. Spark GraphX allows graph-based semantic web data processing. The fundamental objective of this work is to provide a way for efficiently combining Semantic Web and big data technologies to utilize their combined strengths in data analysis and processing. First, the proposed approach uses the SPARQL query language to extract Semantic Web data from DBpedia datasets. DBpedia is a hugely available Semantic Web dataset built on Wikipedia. Secondly, the extracted Semantic Web data was converted to the GraphX data format; vertices and edges files were generated. The conversion process is implemented using Apache Spark GraphX. Third, both vertices and edge tables are stored in HDFS and are available for visualization and analysis operations. Furthermore, the proposed techniques improve the data storage efficiency by reducing the amount of storage space by half when converting from Semantic Web Data to a GraphX file, meaning the RDF size is around 133.8 and GraphX is 75.3. Adopting parallel data processing provided by Apache Spark in the proposed technique reduces the required data processing and analysis time. This article concludes that Apache Spark GraphX can enhance Semantic Web and Big Data technologies. We minimize data size and processing time by converting Semantic Web data to GraphX format, enabling efficient data management and seamless integration.

Enhancing Hadoop distributed storage efficiency using multi-agent systems

<p>Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.</p>

Kafka‐Shield: Kafka Streams‐based distributed detection scheme for IoT traffic‐based DDoS attacks

AbstractWith the rapid proliferation of insecure Internet of Things (IoT) devices, the security of Internet‐based applications and networks has become a prominent concern. One of the most significant security threats encountered in IoT environments is a Distributed Denial of Service (DDoS) attack. This attack can severely disrupt critical services and prevent smart devices from functioning normally, leading to severe consequences for businesses and individuals. It aims to overwhelm victims' resources, websites, and other services by flooding them with massive attack packets, making them inaccessible to legitimate users. Researchers have developed multiple detection schemes to detect DDoS attacks. As technology advances and other facilitating factors have increased, it is a challenge to identify such powerful attacks in real‐time. In this paper, we propose a novel distributed detection scheme for IoT network traffic‐based DDoS attacks by deploying it in a Kafka Streams processing framework named Kafka‐Shield. The Kafka‐Shield comprises two stages: design and deployment. Firstly, the detection scheme is designed on the Hadoop cluster employing a highly scalable H2O.ai machine learning platform. Secondly, a portable, scalable, and distributed detection scheme is deployed on the Kafka Streams processing framework. To analyze the incoming traffic data and categorize it into nine target classes in real time. Additionally, Kafka‐Shield stores each network flow with significant input features and the predicted outcome in the Hadoop Distributed File System (HDFS). It enables the development of new models or updating current ones. To validate the effectiveness of the Kafka‐Shield, we performed critical analysis using various configured attack scenarios. The experimental results affirm Kafka‐Shield's remarkable efficiency in detecting DDoS attacks. It has a detection rate of over 99% and can process 0.928 million traces in nearly 3.027 s.

Survey on MapReduce Scheduler Algorithms in Hadoop Framework

Hadoop serves as a robust framework tailored for the storage and processing of vast data volumes across clusters. Its foundation lies in the Hadoop Distributed File System (HDFS) for data storage, complemented by the MapReduce paradigm for data processing. MapReduce, functioning as a parallel computing framework, operates adeptly on distributed clusters to manage large-scale data processing tasks. Within this framework, scheduling emerges as a pivotal aspect, influencing the overall system's performance and efficiency. The essence of scheduling in MapReduce lies in enhancing performance, minimizing response times, and optimizing resource allocation. This paper undertakes a systematic exploration of existing scheduling algorithms, offering a fresh classification and detailed examination of each category. Furthermore, the analysis delves into the core principles, objectives, as well as the strengths and weaknesses of these scheduling algorithms.

Anomaly detection in log-event sequences: A federated deep learning approach and open challenges

Anomaly Detection (AD) is an important area to reliably detect malicious behavior and attacks on computer systems. Log data is a rich source of information about systems and thus provides a suitable input for AD. With the sheer amount of log data available today, for years Machine Learning (ML) and more recently Deep Learning (DL) have been applied to create models for AD. Especially when processing complex log data, DL has shown some promising results in recent research to spot anomalies. It is necessary to group these log lines into log-event sequences, to detect anomalous patterns that span over multiple log lines. This work uses a centralized approach using a Long Short-Term Memory (LSTM) model for AD as its basis which is one of the most important approaches to represent long-range temporal dependencies in log-event sequences of arbitrary length. Therefore, we use past information to predict whether future events are normal or anomalous. For the LSTM model we adapt a state of the art open source implementation called LogDeep. For the evaluation, we use a Hadoop Distributed File System (HDFS) data set, which is well studied in current research. In this paper we show that without padding, which is a commonly used preprocessing step that strongly influences the AD process and artificially improves detection results and thus accuracy in lab testing, it is not possible to achieve the same high quality of results shown in literature. With the large quantity of log data, issues arise with the transfer of log data to a central entity where model computation can be done. Federated Learning (FL) tries to overcome this problem, by learning local models simultaneously on edge devices and overcome biases due to a lack of heterogeneity in training data through exchange of model parameters and finally arrive at a converging global model. Processing log data locally takes privacy and legal concerns into account, which could improve coordination and collaboration between researchers, cyber security companies, etc., in the future. Currently, there are only few scientific publications on log-based AD which use FL. Implementing FL gives the advantage of converging models even if the log data are heterogeneously distributed among participants as our results show. Furthermore, by varying individual LSTM model parameters, the results can be greatly improved. Further scientific research will be necessary to optimize FL approaches.

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments.

With the development of synchrotron radiation sources and high-frame-rate detectors, the amount of experimental data collected at synchrotron radiation beamlines has increased exponentially. As a result, data processing for synchrotron radiation experiments has entered the era of big data. It is becoming increasingly important for beamlines to have the capability to process large-scale data in parallel to keep up with the rapid growth of data. Currently, there is no set of data processing solutions based on the big data technology framework for beamlines. Apache Hadoop is a widely used distributed system architecture for solving the problem of massive data storage and computation. This paper presents a set of distributed data processing schemes for beamlines with experimental data using Hadoop. The Hadoop Distributed File System is utilized as the distributed file storage system, and Hadoop YARN serves as the resource scheduler for the distributed computing cluster. A distributed data processing pipeline that can carry out massively parallel computation is designed and developed using Hadoop Spark. The entire data processing platform adopts a distributed microservice architecture, which makes the system easy to expand, reduces module coupling and improves reliability.

KS-SDN-DDoS: A Kafka streams-based real-time DDoS attack classification approach for SDN environment

Software-Defined Networking (SDN) is a modern networking architecture that segregates control logic from data plane and supports a loosely coupled architecture. It provides flexibility in this advanced networking paradigm for any changes. Further, it controls the complete network in a centralized using controller(s). However, it comes with several security issues: Exhausting bandwidth and flow tables, Distributed Denial of Service (DDoS) attacks, etc. DDoS is a powerful attack for Internet-based applications and services, traditional and SDN paradigms. In the case of the SDN environment, attackers frequently target the central controller(s). This paper proposes a Kafka Streams-based real-time DDoS attacks classification approach for the SDN environment, named KS-SDN-DDoS. The KS-SDN-DDoS has been designed using highly scalable H2O ML techniques on the two-node Apache Hadoop Cluster (AHC). It consists of two modules: (i) Network Traffic Capture (NTCapture) and (ii) Attack Detection and Traffic Classification (ADTClassification). The NTCapture is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-1). It captures incoming network traffic, extracts and formulates attributes, and publishes significant network traffic attributes on the Kafka topic. The ADTClassification is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-2). It consumes network flows from the Kafka topic, classifies it based on the ten attributes, and publishes it to the decision Kafka topic. Further, it saves attributes with outcome to the Hadoop Distributed File System (HDFS). The KS-SDN-DDoS approach is designed and validated using the recent “DDoS Attack SDN dataset”. The result shows that the proposed system gives better classification accuracy (100%).

DATA FINDING, SHARING AND DUPLICATION REMOVAL IN THE CLOUD

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Design of storage benchmark kit framework for supporting the file storage retrieval

An open-source software framework called the storage benchmark kit (SBK) is used to store the system benchmarking performance framework. The SBK is designed to perform any storage client or device using any data type as a payload. SBK simultaneously helps number of readers as well as writes to the storage system of large amounts of data as well as allows end-to-end latency benchmarking for multiple writers and readers. The SBK uses standardized performance measures for comparing and evaluating various storage systems and their combinations. Distributed file systems, distributed database systems, single or local node databases, systems of object storage, platforms of distributed streaming and messaging, and systems of key-value storage are the storage solutions supported by SBK. The SBK supports various storage systems like XFS, Kafka streaming storage systems, and Hadoop distributed file system (HDFS) performance benchmarking. The experimental results show that a proposed method achieves execution time of 65.530 s, 40.826 s and 30.351 s for the 100k, 500k and 1000k files respectively which ensures better improvement than the existing methods such as simple data interface and distributed data protection system.

Social Media-based Polling Data Processing using MapReduce on the Hadoop Framework

The processing of data obtained from a polling system holds significant importance as the results can serve as a reference for addressing community issues. With the escalating use of social media, Indonesia ranks fifth globally in Twitter usage. When dealing with substantial data sets, processing speed becomes a concern, prompting the author to develop a more time-efficient system for handling polling data gathered through social media. Hadoop emerges as a prime candidate for this purpose due to its two main modules: Hadoop Distributed File System (HDFS) for distributed storage and MapReduce for algorithmic computation. Through experimentation using both the wordcount program with MapReduce on Hadoop and the version without MapReduce, it was observed that MapReduce outperforms the latter in terms of data processing speed. On average, employing MapReduce on Hadoop resulted in a 1.3 times faster data processing rate compared to the method without MapReduce.

CLOUD BASED DUPLICATION REMOVAL SYSTEM

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Investigating optimal features in log files for anomaly detection using optimization approach

<p><span lang="EN-US">Logs have been frequently utilised in different software system administration activities. The number of logs has risen dramatically due to the vast scope and complexity of current software systems. A lot of research has been done on log-based anomaly identification using machine learning approach. In this paper, we proposed an optimization approach to select the optimal features from the logs. This will provide the higher classification accuracy on reduced log files. In order to predict the anomalies three phases are used: i) log representation ii) feature selection and iii) Performance evaluation. The efficacy of the proposed model is evaluated using benchmark datasets such as BlueGene/L (BGL), Thunderbird, spirit and hadoop distributed file system (HDFS) in terms of accuracy, converging ability, train and test accuracy, receiver operating characteristic (ROC) measures, precision, recall and F1-score. The results shows that the feature selection on log files outperforms in terms all the evaluation measures.</span></p>

A MapReduce-Based Approach for Fast Connected Components Detection from Large-Scale Networks.

Owing to increasing size of the real-world networks, their processing using classical techniques has become infeasible. The amount of storage and central processing unit time required for processing large networks is far beyond the capabilities of a high-end computing machine. Moreover, real-world network data are generally distributed in nature because they are collected and stored on distributed platforms. This has popularized the use of the MapReduce, a distributed data processing framework, for analyzing real-world network data. Existing MapReduce-based methods for connected components detection mainly struggle to minimize the number of MapReduce rounds and the amount of data generated and forwarded to the subsequent rounds. This article presents an efficient MapReduce-based approach for finding connected components, which does not forward the complete set of connected components to the subsequent rounds; instead, it writes them to the Hadoop Distributed File System as soon as they are found to reduce the amount of data forwarded to the subsequent rounds. It also presents an application of the proposed method in contact tracing. The proposed method is evaluated on several network data sets and compared with two state-of-the-art methods. The empirical results reveal that the proposed method performs significantly better and is scalable to find connected components in large-scale networks.

Securing cloud-enabled smart cities by detecting intrusion using spark-based stacking ensemble of machine learning algorithms

<abstract> <p>With the use of cloud computing, which provides the infrastructure necessary for the efficient delivery of smart city services to every citizen over the internet, intelligent systems may be readily integrated into smart cities and communicate with one another. Any smart system at home, in a car, or in the workplace can be remotely controlled and directed by the individual at any time. Continuous cloud service availability is becoming a critical subscriber requirement within smart cities. However, these cost-cutting measures and service improvements will make smart city cloud networks more vulnerable and at risk. The primary function of Intrusion Detection Systems (IDS) has gotten increasingly challenging due to the enormous proliferation of data created in cloud networks of smart cities. To alleviate these concerns, we provide a framework for automatic, reliable, and uninterrupted cloud availability of services for the network data security of intelligent connected devices. This framework enables IDS to defend against security threats and to provide services that meet the users' Quality of Service (QoS) expectations. This study's intrusion detection solution for cloud network data from smart cities employed Spark and Waikato Environment for Knowledge Analysis (WEKA). WEKA and Spark are linked and made scalable and distributed. The Hadoop Distributed File System (HDFS) storage advantages are combined with WEKA's Knowledge flow for processing cloud network data for smart cities. Utilizing HDFS components, WEKA's machine learning algorithms receive cloud network data from smart cities. This research utilizes the wrapper-based Feature Selection (FS) approach for IDS, employing both the Pigeon Inspired Optimizer (PIO) and the Particle Swarm Optimization (PSO). For classifying the cloud network traffic of smart cities, the tree-based Stacking Ensemble Method (SEM) of J48, Random Forest (RF), and eXtreme Gradient Boosting (XGBoost) are applied. Performance evaluations of our system were conducted using the UNSW-NB15 and NSL-KDD datasets. Our technique is superior to previous works in terms of sensitivity, specificity, precision, false positive rate (FPR), accuracy, F1 Score, and Matthews correlation coefficient (MCC).</p> </abstract>

Introducing an intelligent multi-level retrieval method for mineral resource potential evaluation result data

Abstract The geological data of the mineral resource potential evaluation results (MRPERs) are diverse and extremely large; efficiently retrieving data remains a challenging problem. In this work, a new way of using the Hadoop platform is proposed. The Hadoop distributed file system is used to store the massive data and construct the data storage model of geological and mineral resources. Using a distributed Hadoop database (HBase) that supports the fast query of a single record, it manages its metadata and retrieves the data of MRPERs quickly. At the same time, a multi-level index directory is designed to support the non-main key query on the HBase. This overcomes the shortcoming that the HBase only supports the simple index based on the main key and realizes the intelligent, efficient retrieval of MRPERs. The validity and feasibility of the proposed method are further verified by experiments using the MRPER data in the Institute of Mineral Resources, Chinese Academy of Geological Sciences.

Blockchain Enabled Hadoop Distributed File System Framework for Secure and Reliable Traceability

Hadoop Distributed File System (HDFS) is a distributed file system that allows large amounts of data to be stored and processed across multiple servers in a Hadoop cluster. HDFS also provides high throughput for data access. HDFS enables the management of vast amounts of data using commodity hardware. However, security vulnerabilities in HDFS can be manipulated for malicious purposes. This emphasizes the significance of establishing strong security measures to facilitate file sharing within Hadoop and implementing a reliable mechanism for verifying the legitimacy of shared files. The objective of this paper is to enhance the security of HDFS by utilizing a blockchain-based technique. The proposed model uses the Hyperledger Fabric platform at the enterprise level to leverage metadata of files, thereby establishing dependable security and traceability of data within HDFS. The analysis of results indicates that the proposed model incurs a slightly higher overhead compared to HDFS and requires more storage space. However, this is considered an acceptable trade-off for the improved security.

Cost-based Data Prefetching and Scheduling in Big Data Platforms over Tiered Storage Systems

The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN’s resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency.

A blockchain integration to support failures prediction from log files in multi-agent systems technology

Computer system failures can cause significant financial and reputational damage, making it critical to detect and prevent them before they occur. In this article, we propose a novel approach for predicting system failures based on log file analysis, called MABFailPred. Our approach leverages a Blockchain-based Multi-Agent architecture to analyze execution traces and detect early warning signals for system failure prediction during execution. Moreover, we introduce a new method for extracting log keys called HCLPars. The MABFailPred approach comprises four agents that work together to gather log keys, categorize data, develop a failure model, and make predictions. The use of blockchain ensures that agent and administrator predictions are synchronized, and agent collaboration is coordinated. We validate our approach by applying it to a real-world distributed system and accurately predicting problems before they occur. Our proposed approach represents a significant advancement in the field of system failure prediction, with the potential to enhance the reliability and security of computer systems. By employing a machine learning algorithm, our approach can predict anomalies in real-time and places a strong emphasis on the continuous monitoring of system behavior. The paper includes a validation of the approach using the Hadoop Distributed File System (HDFS) log dataset. According to our findings, the proposed approach achieves superior performance compared to other methods on the same dataset, with a precision of 0.998. This result establishes a new state-of-the-art performance level for the dataset.