Distributed File System Research Articles

A Literature Review of Big Data and Metadata Management Systems

Abstract: Metadata is essential for the successful operation of data files, as it acts as the underlying data that helps maintain and manage the namespace, permissions, and address of the file data blocks. The performance of metadata services has a direct impact on the overall performance of the file system, as these operations can account for a significant portion, up to 80%, of file system activity. Large-scale distributed file systems (DFS) have emerged to meet the rising demand for data storing. These systems provide users with access to numerous storage units spread across various locations. The widespread use of large-scale DFSs as the foundation for a range of computing systems and applications, particularly in the era of Big Data, highlights the importance of efficient metadata management. This paper seeks to offer an enhanced knowledge of the current status of metadata systems in large-scale DFSs, this paper presents a comprehensive overview of the latest research in the field. DFSs are defined by three aspects: scalability, performance, and availability. Moreover, the study evaluates various challenges and limitations in the current research, providing valuable insight for future studies. Overall, the study attempts to give a complete knowledge of the present scenario of metadata services across large-scale DFSs and to recommend areas for future research and development.

Blockchain and Interplanetary File System (IPFS)-Based Data Storage System for Vehicular Networks with Keyword Search Capability

Closed-circuit television (CCTV) cameras and black boxes are indispensable for road safety and accident management. Visible highway surveillance cameras can promote safe driving habits while discouraging moving violations. According to CCTV laws, footage captured by roadside cameras must be securely stored, and authorized persons can access it. Footages collected by CCTV and Blackbox are usually saved to the camera’s microSD card, the cloud, or hard drives locally but there are concerns about security and data integrity. These issues may be addressed by blockchain technology. The cost of storing data on the blockchain, on the other hand, is prohibitively expensive. We can have decentralized and cost-effective storage with the interplanetary file system (IPFS) project. It is a file-sharing protocol that stores and distributes data in a distributed file system. We propose a decentralized IPFS and blockchain-based application for distributed file storage. It is possible to upload various types of files into our decentralized application (DApp), and hashes of the uploaded files are permanently saved on the Ethereum blockchain with the help of smart contracts. Because it cannot be removed, it is immutable. By clicking on the file description, we can also view the file. DApp also includes a keyword search feature to assist us in quickly locating sensitive information. We used Ethers.js’ smart contract event listener and contract.queryFilter to filter and read data from the blockchain. The smart contract events are then written to a text file for our DApp’s keyword search functionality. Our experiment demonstrates that our DApp is resilient to system failure while preserving the transparency and integrity of data due to the immutability of blockchain.

Generative Adversarial Networks (GAN) and HDFS-Based Realtime Traffic Forecasting System Using CCTV Surveillance

The most crucial component of any smart city traffic management system is traffic flow prediction. It can assist a driver in selecting the most efficient route to their destination. The digitalization of closed-circuit television (CCTV) systems has resulted in more effective and capable surveillance imaging systems for security applications. The number of automobiles on the world’s highways has steadily increased in recent decades. However, road capacity has not developed at the same rate, resulting in significantly increasing congestion. The model learning mechanism cannot be guided or improved by prior domain knowledge of real-world problems. In reality, symmetrical features are common in many real-world research objects. To mitigate this severe situation, the researchers chose adaptive traffic management to make intelligent and efficient use of the current infrastructure. Data grow exponentially and become a complex item that must be managed. Unstructured data are a subset of big data that are difficult to process and have volatile properties. CCTV cameras are used in traffic management to monitor a specific point on the roadway. CCTV generates unstructured data in the form of images and videos. Because of the data’s intricacy, these data are challenging to process. This study proposes using big data analytics to transform real-time unstructured data from CCTV into information that can be shown on a web dashboard. As a Hadoop-based architectural stack that can serve as the ICT backbone for managing unstructured data efficiently, the Hadoop Distributed File System (HDFS) stores several sorts of data using the Hadoop file storage system, a high-performance integrated virtual environment (HIVE) tables, and non-relational storage. Traditional computer vision algorithms are incapable of processing such massive amounts of visual data collected in real-time. However, the inferiority of traffic data and the quality of unit information are always symmetrical phenomena. As a result, there is a need for big data analytics with machine learning, which entails processing and analyzing vast amounts of visual data, such as photographs or videos, to uncover semantic patterns that may be interpreted. As a result, smart cities require a more accurate traffic flow prediction system. In comparison to other recent methods applied to the dataset, the proposed method achieved the highest accuracy of 98.21%. In this study, we look at the construction of a secure CCTV strategy that predicts traffic from CCTV surveillance using real-time traffic prediction analysis with generative adversarial networks (GAN) and HDFS.

HRFP: Highly Relevant Frequent Patterns-Based Prefetching and Caching Algorithms for Distributed File Systems

Data-intensive applications are generating massive amounts of data which is stored on cloud computing platforms where distributed file systems are utilized for storage at the back end. Most users of those applications deployed on cloud computing systems read data more often than they write. Hence, enhancing the performance of read operations is an important research issue. Prefetching and caching are used as important techniques in the context of distributed file systems to improve the performance of read operations. In this research, we introduced a novel highly relevant frequent patterns (HRFP)-based algorithm that prefetches content from the distributed file system environment and stores it in the client-side caches that are present in the same environment. We have also introduced a new replacement policy and an efficient migration technique for moving the patterns from the main memory caches to the caches present in the solid-state devices based on a new metric namely the relevancy of the patterns. According to the simulation results, the proposed approach outperformed other algorithms that have been suggested in the literature by a minimum of 15% and a maximum of 53%.

Blockchain-Based Decentralized Architecture for Software Version Control

Version control is an important component of configuration management, and most enterprise-level software uses different tools and technologies to manage the software version control such as CVS, Subversion, or Perforce. Following the success of bitcoin, the first practical application of blockchain, it is being implemented in other fields such as healthcare, supply chains, financial management, real estate, electoral systems, and so on. Blockchain’s core features include decentralization, immutability, and interminability. Most version control repositories are centralized and can be modified by external sources, implying that they are in danger of being corrupted or controlled. In this study, we present the BDA-SCV architecture for implementing a version control system in blockchain technology. Our proposed approach would replace the necessity for a centralized system, with a decentralized approach implemented in the blockchain using distributed file storage, for which we will use the InterPlanetary File System (IPFS), which is a distributed file system. The proof of authority (PoA) consensus algorithm will be used to approve the developer communicating modifications to the private blockchain network; the authority will only provide permission and will not be able to add, edit, or delete code files. For each change, a ledger block will be created with a reference to the file stored in the distributed repository. A block cannot be manipulated once it has been created. Smart contracts will be used to register developers, create blocks, and manage the repository. The suggested model is implemented using the Hyperledger Fabric network, and the developer and authorizer ends are built into the dotnet web application.

Hadoop-based File Monitoring System for Processing Image Data

This paper presents a file monitoring system based on the Hadoop framework, specifically designed for image data processing. The system comprises a Hadoop cluster and a client, where the Hadoop cluster includes various modules such as a name node module, a name node agent module, data node modules, a matching module, and a response algorithm module. The name node agent module acts as an intermediary between the client and the name node module, forwarding function information and acquiring configuration information. The system provides comprehensive monitoring capabilities for the distributed file system, enabling real-time handling of requests and messages.

Microblog Emotion Analysis Using Improved DBN Under Spark Platform

In order to solve the problems that traditional single-machine methods find it difficult to complete the task of emotion classification quickly, and the time efficiency and scalability are not high; a microblog emotion analysis method using improved deep belief network (DBN) under Spark platform is proposed. First, the Hadoop distributed file system is used to realize the distributed storage of text data, and the preprocessed data and emotion dictionary are converted into word vector representation based on the continuous bag-of-words model. Then, an improved DBN model is constructed by combining the adaptive learning method of DBN with the active learning method, and it is applied to the learning analysis of text word vectors. Finally, the data parallel optimization of the improved DBN model is realized, based on Spark platform to accurately and quickly obtain the emotion types of microblog texts. The experimental analysis of the proposed method based on the microblog text data set shows that the classification accuracy is more than 94%.

DATA CLEANING BEFORE UPLOADING TO STORAGE

The article considered the issue of cleaning big data before uploading it to storage. At this time, the errors made and the methods of eliminating these errors have been clarified. The technology of creating a big data storage and analysis system is reviewed, as well as solutions for the implementation of the first stages of the Data Science process: data acquisition, cleaning and loading are described. The results of the research allow us to move towards the realization of future steps in the field of big data processing. It was noted that Data cleansing is an essential step in working with big data, as any analysis based on inaccurate data can lead to erroneous results. Also, it was noted that cleaning and consolidation of data can also be performed when the data is loaded into a distributed file system. The methods of uploading data to the storage system have been tested. An assembly from Hortonworks was used as the implementation. The easiest way to upload is to use the web interface of the Ambari system or to use HDFS commands to upload to HDFS Hadoop from the local system. It has been shown that the ETL process should be considered more broadly than just importing data from receivers, minimal transformations and loading procedures into the warehouse. Data cleaning should become a mandatory stage of work, because the cost of storage is determined not only by the amount of data, but also by the quality of the information collected. Keywords: Big Data, Data Cleaning, Storage System, ETL process, Loading methods.

Efficient I/O Merging Scheme for Distributed File Systems

Recently, decentralized file systems are widely used to overcome centralized file systems’ load asymmetry between nodes and the scalability problem. Due to the lack of a metadata server, decentralized systems require more RPC requests to control metadata processing between clients and servers, which adversely impacts the I/O performance and traffic imbalance by increasing RPC latency. In this paper, we propose an efficient I/O scheme to reduce the RPC overhead in decentralized file systems. Instead of sending a single RPC request at a time, we enqueued the RPCs in the global queue and merged them into larger RPC requests, thus avoiding excessive RPC latency overheads. The experimental results showed that our scheme improves write and read performance by up to 13% and 16%, respectively, compared with those of the original.

Distributed file systems performance tests on Kubernetes/Docker clusters

Modern data centers need distributed file systems to provide user applications with access to data stored on a large number of nodes. The ability to mount a distributed file system and leverage its native application programming interfaces in a Docker container, combined with the advanced orchestration features provided by Kubernetes, can improve flexibility in installing, monitoring and recovering data management and transfer services. At INFN-CNAF some distributed file systems (i.e. IBM Spectrum Scale, CephFS and Lustre-ZFS) deployment tests with Kubernetes and Docker have been conducted recently with positive results. The purpose of this paper is to show the throughput scores of the previously mentioned file systems when their servers are containerized and run on bare metal machines using a container orchestration framework. This is a preliminary study: for the time being, only sequential read/write tests have been considered.

Design of an IoT platform for data analytics based fault detection and classification in solar PV power plants using CFKC and ODENN

New policies are commenced all over the globe to diminish the use of fossil fuels, which gives rise to the augmented utilization of solar energy (SE). The photovoltaic (PV) system’s performance is extremely environmental variables reliant. Long-range transmission of SE is incompetent as well as complex to carry in the PV system. It can be affected by disparate sorts of faults, which cause severe energy loss all through the system operation. Thus, it is vital to incessantly monitor the solar PV (SPV) system to detect as well classify the faults by preventing energy losses. The IoT applications in SE production engage sensor devices that are fixed to the generation, and transmission, together with distribution equipment. These devices assist in monitoring the operation of the SPV power plant (SPVPP) system remotely in real-time. Presenting a new algorithm that can perform fault detection and classification in a PV system to a higher level of accuracy is the major contribution of this work. Thus, this work designs as well as develops an IoT platform for carrying out analytical tasks that can analyze data generated as of IoT operating systems to detect as well as classify faults in the SPVPP. Initially, the data collected from the dataset is pre-processed in which data duplication is performed using Hadoop distributed file system (HDFS) and then the fault is detected from the pre-processed data using the cosine function based k-means clustering (CFKC) technique in the SPV system. Finally, the obtained fault data is fed into the optimized deep learning centered ENN (ODENN) method which classifies the faults. The proposed techniques detect as well as classify the faults effectively that are experimentally proved by means of comparing them with the prevailing techniques, namely ENN, ANN and SVM, along with KNN in terms of some quality measures. The obtained results for ODENN showed an accuracy of 98.99%, specificity of 97.6%, and a sensitivity of 97.02%.

Optimized hadoop map reduce system for strong analytics of cloud big product data on amazon web service

Because of the rapid increase of data in the cloud of Amazon Web Service (AWS), the traditional methods for analyzing this data are not good and inappropriate, so unconventional methods of analysis have been proposed by many data scientists such as concurrent/ parallel techniques to meeting the requirements of performance and scalability entailed in such big data analyses. In this paper we are used Hadoop Map Reduce system that contains Hadoop Distributed File System (HDFS) and Hadoop cluster. We optimized it by combining it with five efficient Data Mining (DM) algorithms such as Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), Correlative Naïve Bayes classifier (CNB), and Fuzzy CNB (FCNB) for strong analytics of cloud big data. The proposed system applied on product review data that taken form the cloud of AWS. The Evaluation of Hadoop Map Reduce done with important benchmarks as Mean Absolute Percentage Error (MPAE), Root Mean Square Error (RMSE), and runtime for word count, sort, inverted index. Also, the evaluation of DM models with Hadoop Map Reduce system done by using accuracy, sensitivity, specificity, memory, and running time. Experiments have shown that FCNB is effective in addressing the problem of big data.

TS-REPLICA: A novel replica placement algorithm based on the entropy weight TOPSIS method in spark for multimedia data analysis

Performance optimization based on node attributes is of profound significance in the replica placement algorithm used in Hadoop distributed file system (HDFS). Currently, most researchers studying replica placement algorithms consider only a single attribute or a few multiple attributes. However, a single attribute cannot accurately express the performance of a node. Therefore, this paper proposes a replica placement algorithm based on the entropy weight TOPSIS (technique for order preference by similarity to ideal solution) method, called TS-REPLICA. First, a multi-attribute matrix that comprehensively reflects the performance and load of nodes is defined. Then, a TOPSIS-based algorithm is proposed to calculate the performance score of each data node. In addition, the entropy weight method is introduced to derive the weights of attributes for balancing the influence of weights of multiple attributes. Next, the comprehensive load score of each data node in the Spark cluster, the average comprehensive load score of each rack, and the average comprehensive load score of the entire cluster are calculated, and replica placement is performed based on the obtained scores. Finally, the effectiveness of the proposed algorithm is verified on various datasets and test cases. The experimental results show that the TS-REPLICA algorithm outperforms the better comparison algorithm in execution number in Spark cluster.

Introducing UWF-ZeekData22: A Comprehensive Network Traffic Dataset Based on the MITRE ATT&CK Framework

With the rapid rate at which networking technologies are changing, there is a need to regularly update network activity datasets to accurately reflect the current state of network infrastructure/traffic. The uniqueness of this work was that this was the first network dataset collected using Zeek and labelled using the MITRE ATT&CK framework. In addition to identifying attack traffic, the MITRE ATT&CK framework allows for the detection of adversary behavior leading to an attack. It can also be used to develop user profiles of groups intending to perform attacks. This paper also outlined how both the cyber range and hadoop’s big data platform were used for creating this network traffic data repository. The data was collected using Security Onion in two formats: Zeek and PCAPs. Mission logs, which contained the MITRE ATT&CK data, were used to label the network attack data. The data was transferred daily from the Security Onion virtual machine running on a cyber range to the big-data platform, Hadoop’s distributed file system. This dataset, UWF-ZeekData22, is publicly available at datasets.uwf.edu.

Dynamic Replication Policy on HDFS Based on Machine Learning Clustering

Data growth in recent years has been swift, leading to the emergence of big data science. Distributed File Systems (DFS) are commonly used to handle big data, like Google File System (GFS), Hadoop Distributed File System (HDFS), and others. The DFS should provide the availability of data and reliability of the system in case of failure. The DFS replicates the files in different locations to provide availability and reliability. These replications consume storage space and other resources. The importance of these files differs depending on how frequently they are used in the system. So some of these files do not deserve to replicate many times because it is unimportant in the system. This paper introduces a Dynamic Replication Policy using Machine Learning Clustering (DRPMLC) on HDFS, which uses Machine Learning to cluster the files into different groups and apply other replication policies to each group to reduce the storage consumption, improve the read and write operations time and keep the availability and reliability of HDFS as a High-Performance Distributed Computing (HPDC).

Novel Approach on sustainable Topic Modelling paradigms for Recognition of health-related topics in a social platform through Distributed file system

social media is one of the most sustainable and prominent features of the Internet and has a vast amount of influence in today’s world. All the latest information regarding every topic is available on social media i.e., social media serves as a sustainable repository of the latest information, particularly Twitter where discourse on every major trending topic takes place and is shown to the user hence it is utilized by many individuals all over the world to keep themselves updated with the latest news occurring everywhere in the world. In Current time due to the pandemic, Twitter also played a key role in relaying updated and accurate information regarding many topics related to health care. Millions of people everyday share their ideas and opinions regarding health-related concepts. To analyze these many tweets which contain billions of users is a challenging task and so to understand the key features or topics discussed by the user’s artificial intelligence has a branch called Natural language processing also called as NLP which can help in analyzing texts from a huge amount of dataset. Topic modeling, which is a part of NLP helps in analyzing healthcare data extracted from twitter which would provide a description of current trending healthcare topics and development of most significant tweets and their content.

Implementation of Big Data Privacy Preservation Technique for Electronic Health Records in Multivendor Environment

Various diagnostic health data formats and standards include both structured and unstructured data. Sensitive information contained in such metadata requires the development of specific approaches that can combine methods and techniques that can extract and reconcile the information hidden in such data. However, when this data needs to be processed and used for other reasons, there are still many obstacles and concerns to overcome. Modern approaches based on machine learning including big data analytics, assist in the information refinement process for later use of clinical evidence. These strategies consist of transforming various data into standard formats in specific scenarios. In fact, in order to conform to these rules, only de-identified diagnostic and personal data may be handled for secondary analysis, especially when information is distributed or transferred across institutions. This paper proposes big data privacy preservation techniques using various privacy functions. This research focused on secure data distribution as well as security access control to revoke the malicious activity or similarity attacks from end-user. The various privacy preservation techniques such as data anonymization, generalization, random permutation, k-anonymity, bucketization, l-diversity with slicing approach have been proposed during the data distribution. The efficiency of system has been evaluated in Hadoop distributed file system (HDFS) with numerous experiments. The results obtained from different experiments show that the computation should be changed when changing k-anonymity and l-diversity. As a result, the proposed system offers greater efficiency in Hadoop environments by reducing execution time by 15% to 18% and provides a higher level of access control security than other security algorithms.

HV-SNSP: A Low-Overhead Data Recovery Method Based on Cross-Checking

The failure of a single unreliable commodity component is very common in large-scale distributed storage systems. In order to ensure the reliability of data in large-scale distributed storage systems, many studies have emerged one after another. Among them, Erasure Codes are widely used in actual storage systems, such as Hadoop Distributed File System (HDFS), which provides high fault-tolerance with lower storage overhead. However, usually the recovery of erasure-coded storage system when encountering node failure results in severe cross-node and cross-rack bandwidth consumption, which affects the efficiency of failure recovery and wastes additional resources. In this paper, we improve the erasure coding storage strategy in distributed storage systems, and propose a low-overhead data recovery method based on cross-checking, namely HV-SNSP. In HV-SNSP, horizontal and vertical cross parity checking is realized by adding RS parity inside the data node, that is, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, k)}}$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', \textrm {}k')}}$ </tex-math></inline-formula> storage architecture. Based on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, \textrm {}k)}}$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', k')}}$ </tex-math></inline-formula> , a low-cost supply node selection strategy, namely SNSP, is designed, and nodes with shorter network distance and lower load are selected to participate in recovery. This strategy can effectively reduce the amount of data transmission, shorten the recovery time, and improve the recovery efficiency. The experimental results show that compared with traditional RS, HV-SNSP can reduce the amount of cross-rack data transmission by 62.5% during data recovery, and can shorten the recovery time by up to 42.41%; Compared with D3, HV-SNSP can reduce the occupation of cross-rack bandwidth by 25% and shorten the recovery time by 36.58%.

STUDY OF BIG DATA BASED PROBLEMS FOR DATA ANONYMIZATION

In order to defeat typical attacks like the similarity attack, the probabilistic inference attack, and others that are possible with anonymized data, we have studied different techniques. To anonymize the data set and disseminate the anonymized data set on a distributed environment without endangering data privacy, a privacy-preserving distributed framework is suggested in most of the techniques. It is possible to achieve a better balance between privacy and data utility, and the data utility is demonstrated in terms of conventional measures. The privacy-preserved data set is also subjected to the application of several classifiers in order to measure the utility of the data When sharing and processing data in a distributed setting or with the Internet of Things, data privacy is a crucial requirement. High communication and computational costs are involved in collaborative privacy-preserving data mining based on secured multiparty computation. Data protection against identity revelation is achieved by the use of data anonymization, a promising technology in the field of privacy-preserving data mining. Anonymization faces significant difficulties, including information loss and frequent attacks that may be made on the anonymized data. Utilizing data mining techniques, data anonymization has recently demonstrated a considerable increase in data value. Still, the methods now in use are ineffective for dealing with attacks. Therefore, a clustering-based anonymization approach that is resistant to similarity attacks and attacks based on inference is suggested in this study. On the Hadoop Distributed File System, the anonymized data is dispersed. The technique creates a better balance between utility and privacy.

Design of Online Vitals Monitor by Integrating Big Data and IoT

In this work, we design a multisensory IoT-based online vitals monitor (hereinafter referred to as the VITALS) to sense four bedside physiological parameters including pulse (heart) rate, body temperature, blood pressure, and peripheral oxygen saturation. Then, the proposed system constantly transfers these signals to the analytics system which aids in enhancing diagnostics at an earlier stage as well as monitoring after recovery. The core hardware of the VITALS includes commercial off-the-shelf sensing devices/medical equipment, a powerful microcontroller, a reliable wireless communication module, and a big data analytics system. It extracts human vital signs in a pre-programmed interval of 30 min and sends them to big data analytics system through the WiFi module for further analysis. We use Apache Kafka (to gather live data streams from connected sensors), Apache Spark (to categorize the patient vitals and notify the medical professionals while identifying abnormalities in physiological parameters), Hadoop Distributed File System (HDFS) (to archive data streams for further analysis and long-term storage), Spark SQL, Hive and Matplotlib (to support caregivers to access/visualize appropriate information from collected data streams and to explore/understand the health status of the individuals). In addition, we develop a mobile application to send statistical graphs to doctors and patients to enable them to monitor health conditions remotely. Our proposed system is implemented on three patients for 7 days to check the effectiveness of sensing, data processing, and data transmission mechanisms. To validate the system accuracy, we compare the data values collected from established sensors with the measured readouts using a commercial healthcare monitor, the Welch Allyn® Spot Check. Our proposed system provides improved care solutions, especially for those whose access to care services is limited.