Distributed File System Research Articles

An archive‐based method for efficiently handling small file problems in HDFS

SummaryHadoop distributed file system (HDFS) performs well when storing and managing large files. However, its performance significantly decreases when dealing with massive small files. In response to this problem, a novel archive‐based solution is proposed. The archive refers to merging multiple small files into larger data files, which can effectively reduce the memory usage of the NameNode. The current archive‐based solutions have the disadvantages of long access time, long archive construction time, and no support for storage, updating and deleting small files in the archive system. Our method utilizes a dynamic hash function to distribute the metadata of small files across multiple metadata files. We construct a primary index that combines dynamic and static indexes for these metadata files. Regarding data files, include some read‐only files and one readable–writable file. A small file's contents are written into a readable and writable file. Upon reaching a predetermined threshold, the readable–writable file transitions into read‐only status, with a fresh readable–writable file replacing it. Experimental results show that the scheme improves the efficiency of archive access and archive creation and is more efficient than the original HDFS storage and update efficiency.

KGFabric: A Scalable Knowledge Graph Warehouse for Enterprise Data Interconnection

Based on the diversified application scenarios at Ant Group, we built the Ant Knowledge Graph Platform (AKGP). It has constructed numerous domain-specific knowledge graphs related to merchants, companies, accounts, products, and more. AKGP manages trillions of structured knowledge graphs, serving search, recommendation, risk control and other businesses. However, as the demand increasing for various workloads such as graph pattern matching, graph representation learning, and cross-domain knowledge reuse, the existing warehouse systems based on relational DBMS or graph databases are unable to meet the requirements. To address these issues, we propose KGFabric, an industrial-scale knowledge graph management system built on the distributed file system (DFS). KGFabric offers a nearline knowledge storage engine that utilizes a Semantic-enhanced Programmable Graph (SPG) model, which is compatible with the Labeled Property Graph (LPG) model. The data is persistently stored in DFS, such as HDFS, which leverages the POSIX file system API, making it suitable for deployment in multi-cloud environment at low cost. KGFabric provides a native graph-based and hybrid storage format that can serve as a shared backend for parallel graph computing systems, significantly accelerating the analysis of multi-workload. Additionally, KGFabric includes a graph fabric framework that minimizes data duplication and guarantees data security. KGFabric is able to manage Peta-scale data and has supported graph fabric and analysis with over 100 billion relations at Ant Group. We conduct experiments on various datasets to evaluate the performance of KGFabric. Compared with popular relational DBMS and graph databases, the storage space for semantic relations is reduced by over 90%. The performance of graph fabric improves by 21× in real-world workloads. In multi-hop semantic graph analysis, KGFabric enhances performance by 100×.

IoT Forensic Cyber Activities Detection and Prevention with Automated Machine Learning Model

The Internet of Things (IoT) has been deployed in a vast range of applications with exponential increases in data size and complexity. Existing forensic techniques are not effective for the accuracy and detection rate of security issues in IoT forensics. Cyber forensic comprises huge volume constraints that are processing huge volumes of data in the Information and Communication Technology (ICT) comprised of IoT devices and platforms. Trust blockchain is effective technology those are utilized to assess the tamper-proof records in all transaction in the IoT environment. With the implementation of trust blockchain the record and transaction are processed with a distributed ledger that is managed by the network nodes. The challenge associated with the trust blockchain in IoT forensics is cost and security. To achieve significant cost-effectiveness organizations, need to evaluate the risks and benefits associated with IoT forensics in the trust blockchain technology. In this paper, developed a Block Chain Enabled Cyber-Physical system with distributed storage. The developed Blockchain model is termed as Integrated Hadoop Blockchain Forensic Machin Learning (IHBF-ML). The IHBF-ML model uses the Hadoop Distributed File System (HDFS) with cyberspace to improve security. Within the IHBF-ML model, IoT data communication is established with the smart contract. The smart contract-based blockchain process uses the Machine Learning model integrated with Cat Boost classification model for anomaly detection. Cost in IoT forensic is minimized with the parallel processing of the data through MapReduce Framework for the traffic translation, extraction, and analysis of the dynamic feature traffic from the IoT environment. The experimental analysis stated that constructed IHBF-ML model reduces the cost by ~25% than the other conventional blockchain Ethereum and EOS.

RADAR-Pipeline: Scalable Feature Generation for Mobile Health Data

Introduction & BackgroundRADAR-Pipeline is an open-source Python framework designed to simplify and enhance mobile health data analysis. It has been designed to efficiently read and process the large amount of data generated through the RADAR-Base platform. RADAR-base is a scalable, real-time streaming and analytics open-source platform to facilitate research access and customisation requirements. Studies using the Radar-base platform have collected fine-grained longitudinal data from wearables and phones. The data can potentially create multitudes of digital biomarkers, which can be used to inform us greatly about the disease condition. Due to the sheer size of the data, it can be difficult for researchers to read and process those data -- a common task is identifying useful features and common data processing/analysis steps previously used by the community. Up to now, these have been hand-crafted by individual data scientists, often lacking the capability to be easily reused by the community without author-specific knowledge. Furthermore, generating variables based on already established research on a larger scale can be challenging and could hinder replication. Hence, we have designed RADAR-Pipeline to help researchers overcome these challenges. It empowers them to create and share their data analysis and visualisation pipelines, fostering collaboration and knowledge sharing within the research community. Objectives & ApproachThe primary objective of RADAR-Pipeline is to offer researchers a user-friendly and powerful platform to develop and share their research. Researchers can build reusable analysis and visualisation pipelines to ensure consistent and reliable results. It simplifies big data analysis by leveraging Apache Spark to handle large and complex mobile health datasets efficiently. Researchers can also save time and effort by reusing and extending existing pipelines built by others. Finally, the RADAR-Pipeline promotes collaboration and recognition by allowing researchers to share their work through the RADAR-base Analytics Catalogue, making their pipelines citable and accessible to the wider research community. Whilst Radar-pipeline has been designed to read data from Radar-base, it can also be used to read data from any dataset which uses Hadoop Distributed File System (HDFS) file system namespace. Relevance to Digital FootprintsMobile health data is rich and valuable for understanding human behaviour and health. RADAR-Pipeline addresses the challenges associated with analysing large and complex mobile health datasets, enabling researchers to extract valuable insights that can be used to (1) Improve public health: By enabling efficient analysis of large-scale mobile health data, RADAR-Pipeline can contribute to research efforts aimed at improving population health outcomes and developing effective interventions; (2) Personalised healthcare: By facilitating the extraction of individual-level features from mobile health data, RADAR-Pipeline can seamlessly be integrated with Kafka data streams and machine learning pipelines to process the data in real-time, which can then be utilised to create more effective and targeted real-time interventions. (3) Promote reproducible research: The framework's emphasis on transparency and reproducibility in research aligns with the conference's focus on the responsible use of digital mobile health data. Conclusions & ImplicationsRADAR-Pipeline is a valuable tool for researchers, offering them the means to harness the potential of mobile health data. By adopting this framework, researchers can achieve efficient and scalable data analysis, thereby streamlining the extracting insights from digital footprints. This efficiency enables researchers to delve deeper into the data and uncover valuable patterns and trends. Furthermore, RADAR-Pipeline promotes collaboration and knowledge sharing within the research community. By providing a standardised framework for data analysis, RADAR-Pipeline facilitates collaboration among researchers, leading to the sharing of best practices and the dissemination of knowledge.

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

The volume of data is growing at an astonishingly high speed. Traditional techniques for storing and processing data, such as relational and centralized databases, have become inefficient and time-consuming. Linked data and the Semantic Web make internet data machine-readable. Because of the increasing volume of linked data and Semantic Web data, storing and working with them using traditional approaches is not enough, and this causes limited hardware resources. To solve this problem, storing datasets using distributed and clustered methods is essential. Hadoop can store datasets because it can use many hard disks for distributed data clustering; Apache Spark can be used for parallel data processing more efficiently than Hadoop MapReduce because Spark uses memory instead of the hard disk. Semantic Web data has been stored and processed in this paper using Apache Spark GraphX and the Hadoop Distributed File System (HDFS). Spark's in-memory processing and distributed computing enable efficient data analysis of massive datasets stored in HDFS. Spark GraphX allows graph-based semantic web data processing. The fundamental objective of this work is to provide a way for efficiently combining Semantic Web and big data technologies to utilize their combined strengths in data analysis and processing. First, the proposed approach uses the SPARQL query language to extract Semantic Web data from DBpedia datasets. DBpedia is a hugely available Semantic Web dataset built on Wikipedia. Secondly, the extracted Semantic Web data was converted to the GraphX data format; vertices and edges files were generated. The conversion process is implemented using Apache Spark GraphX. Third, both vertices and edge tables are stored in HDFS and are available for visualization and analysis operations. Furthermore, the proposed techniques improve the data storage efficiency by reducing the amount of storage space by half when converting from Semantic Web Data to a GraphX file, meaning the RDF size is around 133.8 and GraphX is 75.3. Adopting parallel data processing provided by Apache Spark in the proposed technique reduces the required data processing and analysis time. This article concludes that Apache Spark GraphX can enhance Semantic Web and Big Data technologies. We minimize data size and processing time by converting Semantic Web data to GraphX format, enabling efficient data management and seamless integration.

Enhancing Hadoop distributed storage efficiency using multi-agent systems

<p>Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.</p>

Data Lake Conceptualized Web Platform for Food Research Data Collection

Food research is uniquely intertwined with everyday life and necessitates the utilization of big data. Within this domain, the research data consist of various forms and formats, encompassing biological experiment results, chemical analysis data, nutritional information, microbiological data, sensor data, images, and videos. This diversity stems from the integration of data from various subdomains within the larger field. With recent advancements in deep learning technology, the importance of data has grown significantly, resulting in increased reliance on data-driven research. Although specialized platforms for sharing and utilizing data have been established at the national level, particularly in the bioscience field, food research lacks a dedicated infrastructure and specialized data-sharing platforms. In this study, we develop a platform that leverages Hadoop-based distributed file systems to create a data lake. This platform enables data storage and sharing through a web-based interface. The distributed file system supports scalability by adding data nodes, making it an effective solution for capacity expansion. In addition, the web-based platform ensures high accessibility, allowing users access from anywhere, at any time, using any device. Finally, we introduce the establishment of a 1.8 PB Hadoop-based physical storage system and present an approach for building a highly accessible web platform with substantial utility.

Fractional social optimization-based migration and replica management algorithm for load balancing in distributed file system for cloud computing

ABSTRACT Effective management of data is a major issue in Distributed File System (DFS), like the cloud. This issue is handled by replicating files in an effective manner, which can minimize the time of data access and elevate the data availability. This paper devises a Fractional Social Optimization Algorithm (FSOA) for replica management along with balancing load in DFS in the cloud stage. Balancing the workload for DFS is the main objective. Here, the chunk creation is done by partitioning the file into a different number of chunks considering Deep Fuzzy Clustering (DFC) and then in the round-robin manner the Virtual machine (VM) is assigned. In that case for balancing the load considering certain objectives like resource use, energy consumption and migration cost thereby the load balancing is performed with the proposed FSOA. Here, the FSOA is formulated by uniting the Social optimization algorithm (SOA) and Fractional Calculus (FC). The replica management is done in DFS using the proposed FSOA by considering the various objectives. The FSOA has the smallest load of 0.299, smallest cost of 0.395, smallest energy consumption of 0.510, smallest overhead of 0.358, and smallest throughput of 0.537.

Aguhyper: a hyperledger-based electronic health record management framework

The increasing importance of healthcare records, particularly given the emergence of new diseases, emphasizes the need for secure electronic storage and dissemination. With these records dispersed across diverse healthcare entities, their physical maintenance proves to be excessively time-consuming. The prevalent management of electronic healthcare records (EHRs) presents inherent security vulnerabilities, including susceptibility to attacks and potential breaches orchestrated by malicious actors. To tackle these challenges, this article introduces AguHyper, a secure storage and sharing solution for EHRs built on a permissioned blockchain framework. AguHyper utilizes Hyperledger Fabric and the InterPlanetary Distributed File System (IPFS). Hyperledger Fabric establishes the blockchain network, while IPFS manages the off-chain storage of encrypted data, with hash values securely stored within the blockchain. Focusing on security, privacy, scalability, and data integrity, AguHyper’s decentralized architecture eliminates single points of failure and ensures transparency for all network participants. The study develops a prototype to address gaps identified in prior research, providing insights into blockchain technology applications in healthcare. Detailed analyses of system architecture, AguHyper’s implementation configurations, and performance assessments with diverse datasets are provided. The experimental setup incorporates CouchDB and the Raft consensus mechanism, enabling a thorough comparison of system performance against existing studies in terms of throughput and latency. This contributes significantly to a comprehensive evaluation of the proposed solution and offers a unique perspective on existing literature in the field.

Kafka‐Shield: Kafka Streams‐based distributed detection scheme for IoT traffic‐based DDoS attacks

AbstractWith the rapid proliferation of insecure Internet of Things (IoT) devices, the security of Internet‐based applications and networks has become a prominent concern. One of the most significant security threats encountered in IoT environments is a Distributed Denial of Service (DDoS) attack. This attack can severely disrupt critical services and prevent smart devices from functioning normally, leading to severe consequences for businesses and individuals. It aims to overwhelm victims' resources, websites, and other services by flooding them with massive attack packets, making them inaccessible to legitimate users. Researchers have developed multiple detection schemes to detect DDoS attacks. As technology advances and other facilitating factors have increased, it is a challenge to identify such powerful attacks in real‐time. In this paper, we propose a novel distributed detection scheme for IoT network traffic‐based DDoS attacks by deploying it in a Kafka Streams processing framework named Kafka‐Shield. The Kafka‐Shield comprises two stages: design and deployment. Firstly, the detection scheme is designed on the Hadoop cluster employing a highly scalable H2O.ai machine learning platform. Secondly, a portable, scalable, and distributed detection scheme is deployed on the Kafka Streams processing framework. To analyze the incoming traffic data and categorize it into nine target classes in real time. Additionally, Kafka‐Shield stores each network flow with significant input features and the predicted outcome in the Hadoop Distributed File System (HDFS). It enables the development of new models or updating current ones. To validate the effectiveness of the Kafka‐Shield, we performed critical analysis using various configured attack scenarios. The experimental results affirm Kafka‐Shield's remarkable efficiency in detecting DDoS attacks. It has a detection rate of over 99% and can process 0.928 million traces in nearly 3.027 s.

Tele-Trafficking of Virtual Data Storage Obtained from Smart Grid by Replicated Gluster in Syntose Environment

One of the most important developments in the energy industry is the evolution of smart grids, which record minute details of voltage levels, energy usage, and other critical electrical variables through General Packet Radio Service (GPRS)-enabled meters. This phenomenon creates an extensive dataset for the optimization of the grid system. However, the minute-by-minute energy details recorded by GPRS meters are challenging to store and manage in physical storage resources (old techniques lead to a memory shortage). This study investigates using the distributed file system, replicated Gluster, as a reliable storage option for handling and protecting the enormous volumes of data produced by smart grid components. This study performs two essential tasks. (1) The storage of virtual data received from GPRS meters and load flow analysis of SynerGee Electric 4.0 software from the smart grid (we have extracted electrical data from 16 outgoing feeders, distributed lines, in this manuscript). (2) Tele-trafficking is performed to check the performance of replicated Gluster (RG) for virtual data (electrical data received from the smart grid) storage in terms of User Datagram Protocol (UDP), Transmission Control Protocol (TCP), data flow, and jitter delays. This storage technique provides more opportuni11ty to analyze and perform smart techniques efficiently for future requirement, analysis, and load estimation in smart grids compared to traditional storage methods.

Survey on MapReduce Scheduler Algorithms in Hadoop Framework

Hadoop serves as a robust framework tailored for the storage and processing of vast data volumes across clusters. Its foundation lies in the Hadoop Distributed File System (HDFS) for data storage, complemented by the MapReduce paradigm for data processing. MapReduce, functioning as a parallel computing framework, operates adeptly on distributed clusters to manage large-scale data processing tasks. Within this framework, scheduling emerges as a pivotal aspect, influencing the overall system's performance and efficiency. The essence of scheduling in MapReduce lies in enhancing performance, minimizing response times, and optimizing resource allocation. This paper undertakes a systematic exploration of existing scheduling algorithms, offering a fresh classification and detailed examination of each category. Furthermore, the analysis delves into the core principles, objectives, as well as the strengths and weaknesses of these scheduling algorithms.

Anomaly detection in log-event sequences: A federated deep learning approach and open challenges

Anomaly Detection (AD) is an important area to reliably detect malicious behavior and attacks on computer systems. Log data is a rich source of information about systems and thus provides a suitable input for AD. With the sheer amount of log data available today, for years Machine Learning (ML) and more recently Deep Learning (DL) have been applied to create models for AD. Especially when processing complex log data, DL has shown some promising results in recent research to spot anomalies. It is necessary to group these log lines into log-event sequences, to detect anomalous patterns that span over multiple log lines. This work uses a centralized approach using a Long Short-Term Memory (LSTM) model for AD as its basis which is one of the most important approaches to represent long-range temporal dependencies in log-event sequences of arbitrary length. Therefore, we use past information to predict whether future events are normal or anomalous. For the LSTM model we adapt a state of the art open source implementation called LogDeep. For the evaluation, we use a Hadoop Distributed File System (HDFS) data set, which is well studied in current research. In this paper we show that without padding, which is a commonly used preprocessing step that strongly influences the AD process and artificially improves detection results and thus accuracy in lab testing, it is not possible to achieve the same high quality of results shown in literature. With the large quantity of log data, issues arise with the transfer of log data to a central entity where model computation can be done. Federated Learning (FL) tries to overcome this problem, by learning local models simultaneously on edge devices and overcome biases due to a lack of heterogeneity in training data through exchange of model parameters and finally arrive at a converging global model. Processing log data locally takes privacy and legal concerns into account, which could improve coordination and collaboration between researchers, cyber security companies, etc., in the future. Currently, there are only few scientific publications on log-based AD which use FL. Implementing FL gives the advantage of converging models even if the log data are heterogeneously distributed among participants as our results show. Furthermore, by varying individual LSTM model parameters, the results can be greatly improved. Further scientific research will be necessary to optimize FL approaches.

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments.

With the development of synchrotron radiation sources and high-frame-rate detectors, the amount of experimental data collected at synchrotron radiation beamlines has increased exponentially. As a result, data processing for synchrotron radiation experiments has entered the era of big data. It is becoming increasingly important for beamlines to have the capability to process large-scale data in parallel to keep up with the rapid growth of data. Currently, there is no set of data processing solutions based on the big data technology framework for beamlines. Apache Hadoop is a widely used distributed system architecture for solving the problem of massive data storage and computation. This paper presents a set of distributed data processing schemes for beamlines with experimental data using Hadoop. The Hadoop Distributed File System is utilized as the distributed file storage system, and Hadoop YARN serves as the resource scheduler for the distributed computing cluster. A distributed data processing pipeline that can carry out massively parallel computation is designed and developed using Hadoop Spark. The entire data processing platform adopts a distributed microservice architecture, which makes the system easy to expand, reduces module coupling and improves reliability.

KS-SDN-DDoS: A Kafka streams-based real-time DDoS attack classification approach for SDN environment

Software-Defined Networking (SDN) is a modern networking architecture that segregates control logic from data plane and supports a loosely coupled architecture. It provides flexibility in this advanced networking paradigm for any changes. Further, it controls the complete network in a centralized using controller(s). However, it comes with several security issues: Exhausting bandwidth and flow tables, Distributed Denial of Service (DDoS) attacks, etc. DDoS is a powerful attack for Internet-based applications and services, traditional and SDN paradigms. In the case of the SDN environment, attackers frequently target the central controller(s). This paper proposes a Kafka Streams-based real-time DDoS attacks classification approach for the SDN environment, named KS-SDN-DDoS. The KS-SDN-DDoS has been designed using highly scalable H2O ML techniques on the two-node Apache Hadoop Cluster (AHC). It consists of two modules: (i) Network Traffic Capture (NTCapture) and (ii) Attack Detection and Traffic Classification (ADTClassification). The NTCapture is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-1). It captures incoming network traffic, extracts and formulates attributes, and publishes significant network traffic attributes on the Kafka topic. The ADTClassification is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-2). It consumes network flows from the Kafka topic, classifies it based on the ten attributes, and publishes it to the decision Kafka topic. Further, it saves attributes with outcome to the Hadoop Distributed File System (HDFS). The KS-SDN-DDoS approach is designed and validated using the recent “DDoS Attack SDN dataset”. The result shows that the proposed system gives better classification accuracy (100%).

Distributed File Systems for Collaborative Data Access and Scalability

Distributed File Systems for Collaborative Data Access and Scalability

Distributed matrix computing system for big data

In order to solve the problem of low computing efficiency in big data analysis and model construction, this paper intended to deeply explore the big data analysis programming model, DAG (Directed Acyclic Graph) and other contents, and on this basis, it adopted a distributed matrix computing system Octopus for big data analysis. Octopus is a universal matrix programming framework that provides a programming model based on matrix operations, which can conveniently analyze and process large-scale data. By using Octopus, users can extract functions and data from multiple platforms and operate through a unified matrix operation interface. The distributed matrix representation and storage layer can design data storage formats for distributed file systems. Each computing platform in OctMatrix provides its own matrix library, and it provides a matrix library written in R language for the above users. SymboMatrix provides a matrix interface to OctMatrix that is consistent with OctMatrix. However, SymboMatrix also retains the flow diagram for matrix operations in the process, and it also supports logical and physical optimization of the flow diagram on a DAG. For the DAG computational flow graph generated by SymbolMatrix, this paper divided it into two parts: logical optimization and physical optimization. This paper adopted a distributed file system based on line matrix, and obtained the corresponding platform matrix by reading the documents based on line matrix. In the evaluation of system performance, it was found that the distributed matrix computing system had a high computing efficiency, and the average CPU (central processing unit) usage reached 70%. This system can make full use of computing resources and realize efficient parallel computing.

DATA FINDING, SHARING AND DUPLICATION REMOVAL IN THE CLOUD

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Design of storage benchmark kit framework for supporting the file storage retrieval

An open-source software framework called the storage benchmark kit (SBK) is used to store the system benchmarking performance framework. The SBK is designed to perform any storage client or device using any data type as a payload. SBK simultaneously helps number of readers as well as writes to the storage system of large amounts of data as well as allows end-to-end latency benchmarking for multiple writers and readers. The SBK uses standardized performance measures for comparing and evaluating various storage systems and their combinations. Distributed file systems, distributed database systems, single or local node databases, systems of object storage, platforms of distributed streaming and messaging, and systems of key-value storage are the storage solutions supported by SBK. The SBK supports various storage systems like XFS, Kafka streaming storage systems, and Hadoop distributed file system (HDFS) performance benchmarking. The experimental results show that a proposed method achieves execution time of 65.530 s, 40.826 s and 30.351 s for the 100k, 500k and 1000k files respectively which ensures better improvement than the existing methods such as simple data interface and distributed data protection system.

Strengthening data integrity in academic document recording with blockchain and InterPlanetary file system

A diploma is a certificate or official document given by a school or college that is useful for continuing education, applying for jobs, and assessing student intelligence. The main problem with diplomas and other academic documents is that many are forged. This study aims to develop a prototype for recording student academic data using blockchain and blockchain and InterPlanetary file system (IPFS). The research stages were conducted with system conceptualization, data modeling, smart contract development, IPFS integration, data transaction development, user interface/user experience (UI/UX) development, and system testing. A blockchain is a permanent information structure formed by data blocks that are interconnected with transaction data blocks before and after it. The transaction data for each block are encrypted using asymmetric cryptography. IPFS is a peer-to-peer network protocol for storing and sharing data in a distributed file system applying the concept of decentralization to make the manipulation more difficult. The results show that student academic data and documents were successfully stored in a blockchain network using smart contracts and IPFS. Blockchain technology, smart contracts, and IPFS strengthen the value of these documents into documents that are safe, difficult to counterfeit, and easy to trace, such that authentication and integration are better preserved.