Distributed Storage Architecture Research Articles

Random Partition Based Adaptive Distributed Kernelized SVM for Big Data

In this paper, we present a distributed classification technique for big data by efficiently using distributed storage architecture and data processing units of a cluster. While handling such large data, the existing approaches consider specific data partitioning techniques which demand complete data be processed before partitioning. This leads to an excessive overhead of high computation and data communication. The proposed method does not require any pre-structured data partitioning technique and is also adaptive to big data mining tools. We hypothesize that an effective aggregation of the information generated from data partitions by subprocesses of the complete learning process can lead to accurate prediction results while reducing the overall time complexity. We build three SVM based classifiers, namely one phase voting SVM (1PVSVM), two phase voting SVM (2PVSVM), and similarity based SVM (SIMSVM). Each of these classifiers utilizes the support vectors as the local information to construct the synthesized learner for efficiently reducing the training time and ensuring minimal communication between processing units. In this context, an extensive empirical analysis demonstrates the effectiveness of our classifiers when compared to other existing approaches on several benchmark datasets. However, among existing methods and three of our proposed (1PVSVM, 2PVSIM, and SIMSVM) methods, SIMSVM is the most efficient. Considering MNIST dataset, SIMSVM achieves an average speedup ratio of 0.78 and minimum scalability of 73% when the data size is scaled up to 10 times. It also retains high accuracy (99%) similar to centralized approaches.

Trustworthy Edge Storage Orchestration in Intelligent Transportation Systems Using Reinforcement Learning

A large scale fast-growing data generated in intelligent transportation systems (ITS) has become a ponderous burden on the coordination of heterogeneous transportation networks, which makes the traditional cloud-centric storage architecture no longer satisfy new data analytics requirements. Meanwhile, the lack of storage trust between ITS devices and edge servers could lead to security risks in the data storage process. However, a unified data distributed storage architecture for ITS with intelligent management and trustworthiness is absent in the previous works. To address these challenges, this paper proposes a distributed trustworthy storage architecture with reinforcement learning in ITS, which also promotes edge services. We adopt an intelligent storage scheme to store data dynamically with reinforcement learning based on trustworthiness and popularity, which improves resource scheduling and storage space allocation. Besides, trapdoor hashing based identity authentication protocol is proposed to secure transportation network access. Due to the interaction between cooperative devices, our proposed trust evaluation mechanism is provided with extensibility in the various ITS. Simulation results demonstrate that our proposed distributed trustworthy storage architecture outperforms the compared ones in terms of trustworthiness and efficiency.

Enhanced Secured Communication Optimization In Data Packets Using Proxy Protocols

The principle objective of this paper is to build up a Computational Private Information Retrieval conventions on cloud design utilizing deletion code for made sure about information sending. These conventions are too costly in view of the fact that they are combining complicated numerical operations for the whole database. Essentially distributed storage engineering would provide a range of higher performing capacity staff that provides long-term internet storage management and even the distributed storage architecture. Putting information away and retrieving it in an external cloud system and public review plan creates difficulties and disputes about classification of information during transfers. On any outsider's case, this dispute is usually a massive amount of knowledge stored with the cloud worker. Also you can defeat this problem by means of various methods such as encryption, key encryption, etc. In either case, overall encryption plans maintain confidentiality of information during the exchange, thus restricting the utility of the capability system in parallel with this cycle. This is because a few activities have only been supported by scrambled knowledge. These techniques will cause disappointment. To building a protected stockpiling framework that underpins different capacities is testing when the capacity framework is appropriated and has no focal power.

A distributed multiple sample testing for massive data

ABSTRACT When the data are stored in a distributed manner, direct application of traditional hypothesis testing procedures is often prohibitive due to communication costs and privacy concerns. This paper mainly develops and investigates a distributed two-node Kolmogorov–Smirnov hypothesis testing scheme, implemented by the divide-and-conquer strategy. In addition, this paper also provides a distributed fraud detection and a distribution-based classification for multi-node machines based on the proposed hypothesis testing scheme. The distributed fraud detection is to detect which node stores fraud data in multi-node machines and the distribution-based classification is to determine whether the multi-node distributions differ and classify different distributions. These methods can improve the accuracy of statistical inference in a distributed storage architecture. Furthermore, this paper verifies the feasibility of the proposed methods by simulation and real example studies.

Optimized dynamic storage of data (ODSD) in IoT based on blockchain for wireless sensor networks

The Industry 4.0 IoT network integration with blockchain architecture is a decentralized, distributed ledger mechanism used to record multi-user transactions. Blockchain requires a data storage system designed to be secure, reliable, and fully transparent, emerged as a preferred IoT-based digital storage on WSN. Blockchain technology is being used in the paper to construct the node recognition system according to the storage of data for WSNs. The data storage process on such data must be secure and traceable in different forensics and decision making. The primary theme of the dynamic data security is therefore for rejecting exploitation of the unauthorized user and for evaluating the mechanism in tracing and evidence of system’s data operation in a dynamic manner, growth and quality features under the stochastic state of the model; (1) a mathematical method for the secured storage of data in dynamic is built through distributed node cooperation in IoT industry. (2) the ownership transition feature and the dynamic storage of data system architecture are configured, (3) the emerging distributed storage architecture for blockchain-based WSN will substantially reduce overhead storage for each node without affecting data integrity; (4) minimize the latency of data reconstruction in distributed over storage system, and propose an effective and scalable algorithm for optimizing storage latency issue. In addition to this research, the system implements verified possession of data for replacing the evidence in original digital currency for mining and to store new data blocks that will be compared to the proof system, dramatically reduces computational capacity. The proposed ODSD framework has exceptional benefits for real-time applications while maintaining the security of the dynamic storage of data.

A Cloud-Based Platform for Big Data-Driven CPS Modeling of Robots

This paper proposes an improved cyber-physical systems (CPS) architecture for a smart robotic factory based on an industrial cloud platform driven by big data based on the traditional CPS architecture. This paper uses the architecture analysis and design language to model and design a total of three scales for the underlying cell-level robot, the system-level robot shop, and the overall robotic smart factory CPS, respectively, to complete the conceptual scheme for building a robotic smart factory from a local to an overall CPS system. Using the advantages of cloud computing and combining robotic CPS with cloud computing, an architecture for an industrial management system for CPS cloud computing is proposed. Base based distributed storage architecture with Storm based distributed real-time processing architecture. In terms of modeling, the advantages and disadvantages of using AADL, structural analysis, and design language, and modelers, a physical device modeling language, are combined to analyze the advantages and disadvantages of architecture analysis & design language (AADL) for modeling CPS and propose a CPS analysis and design based on AADL and applicable to it. The paper also investigates the use of LeNet models for state identification in the HSV color space. The algorithm was verified on a self-built power equipment indicator dataset with a 100% detection rate and 99.8% state recognition accuracy after four consecutive frames of fusion detection. Simulink simulation of the trolley was carried out in terms of a cell-level robotic trolley CPS system to demonstrate the effectiveness of the design of a robotic CPS system driven by soaring data based on the industrial cloud platform proposed in this paper.

Sustainable Rural Electrification Through Solar PV DC Microgrids—An Architecture-Based Assessment

Solar photovoltaic (PV) direct current (DC) microgrids have gained significant popularity during the last decade for low cost and sustainable rural electrification. Various system architectures have been practically deployed, however, their assessment concerning system sizing, losses, and operational efficiency is not readily available in the literature. Therefore, in this research work, a mathematical framework for the comparative analysis of various architectures of solar photovoltaic-based DC microgrids for rural applications is presented. The compared architectures mainly include (a) central generation and central storage architecture, (b) central generation and distributed storage architecture, (c) distributed generation and central storage architecture, and (d) distributed generation and distributed storage architecture. Each architecture is evaluated for losses, including distribution losses and power electronic conversion losses, for typical power delivery from source end to the load end in the custom village settings. Newton–Raphson method modified for DC power flow was used for distribution loss analysis, while power electronic converter loss modeling along with the Matlab curve-fitting tool was used for the evaluation of power electronic losses. Based upon the loss analysis, a framework for DC microgrid components (PV and battery) sizing was presented and also applied to the various architectures under consideration. The case study results show that distributed generation and distributed storage architecture with typical usage diversity of 40% is the most feasible architecture from both system sizing and operational cost perspectives and is 13% more efficient from central generation and central storage architecture for a typical village of 40 houses. The presented framework and the analysis results will be useful in selecting an optimal DC microgrid architecture for future rural electrification implementations.

Transaction Data Management System based on Distributed Storage Architecture

With the continuous development of application requirements and technologies, the differences between distributed storage architectures and traditional storage architectures are becoming increasingly apparent. The distributed storage architecture not only has the advantages of small fault impact range, high scalability, and large throughput, but also is widely used in many application systems. In the transaction data management system based on the distributed storage architecture, a large amount of transaction data information is transmitted and stored using the distributed storage architecture, which can effectively ensure the accuracy and consistency of the data. At the same time, the stability of the transaction data management system will be enhanced.

DESIGN AND VERIFICATION OF REMOTE SENSING IMAGE DATA CENTER STORAGE ARCHITECTURE BASED ON HADOOP

Abstract. The data center is a new concept of data processing and application proposed in recent years. It is a new method of processing technologies based on data, parallel computing, and compatibility with different hardware clusters. While optimizing the data storage management structure, it fully utilizes cluster resource computing nodes and improves the efficiency of data parallel application. This paper used mature Hadoop technology to build a large-scale distributed image management architecture for remote sensing imagery. Using MapReduce parallel processing technology, it called many computing nodes to process image storage blocks and pyramids in the background to improve the efficiency of image reading and application and sovled the need for concurrent multi-user high-speed access to remotely sensed data. It verified the rationality, reliability and superiority of the system design by testing the storage efficiency of different image data and multi-users and analyzing the distributed storage architecture to improve the application efficiency of remote sensing images through building an actual Hadoop service system.

Low latency network and distributed storage for next generation HPC systems: the ExaNeSt project

With processor architecture evolution, the HPC market has undergone a paradigm shift. The adoption of low-cost, Linux-based clusters extended the reach of HPC from its roots in modelling and simulation of complex physical systems to a broader range of industries, from biotechnology, cloud computing, computer analytics and big data challenges to manufacturing sectors. In this perspective, the near future HPC systems can be envisioned as composed of millions of low-power computing cores, densely packed — meaning cooling by appropriate technology — with a tightly interconnected, low latency and high performance network and equipped with a distributed storage architecture. Each of these features — dense packing, distributed storage and high performance interconnect — represents a challenge, made all the harder by the need to solve them at the same time. These challenges lie as stumbling blocks along the road towards Exascale-class systems; the ExaNeSt project acknowledges them and tasks itself with investigating ways around them.

Time Series Data Storage System Based on Big Data Technologies: Architecture and Implementation

AbstractA steady usage growth of various smart meters/sensors types comes along with the development of Smart Grid. Due to that smart meters/sensors usually produce data continuously, the state-of-the-art storage architecture will no longer be sustainable under such a dramatic data increase circumstance. It is crucial that these data shall be stored for both short and long term analysis to seek valuable information. This paper proposed a scalable and distributed storage architecture for time series data, and further substantiated the proposed storage framework on a proof-of-concept testbed using a cluster of low-cost and credit-card-sized single-board computers. We evaluated the proposed distributed storage framework and proof-of-concept testbed with a comprehensive set of performance measures. Results showed that low-cost hardware and open source technologies could be utilized for large-scale time series data storage.

A novel HBase data storage in wireless sensor networks

Based on the characteristics of data storage structure of HBase, and under the guidance of the storage optimization strategy, an efficient real-time storage model is designed. The real-time flow of data, which has fast storage in the cluster database, will satisfy the user’s multiple needs of the data storage performance. Extenics primitives are used to integrate heterogeneous data-sets stored in the HBase database, cross-regional wireless sensor network data, and global data storage management catalogs for a double layer distributed storage architecture to improve storage and access efficiency. Under the circumstances that the storage space is insufficient or the storage space of the cluster text system is too massive, the HBase database cluster dynamic updating and optimizing the database space are also the research direction of this paper. Finally, the article carries on the experiment and concludes the appraisal through simulation.

Traffic Analysis for Storage Finding in Video on Demand System

The literature survey typically predicated sharp growth for IP-based video traffic i.e., 30% or more annually. For the Internet TV in mobile networks, video traffic growth rate is expected to rise 80% or more. These high growth rates of video traffic will account for a large portion of the bandwidth. The performance of video-on-demand system during real-time data streaming greatly depends on the session oriented data-storage finding in the mass scale distributed storage architecture. At the storage end, data is broken up into manageable chunks of data packets, which could be smoothly, deliver over the Internet. The objective of this study was to present the necessity of traffic control and traffic analysis methodology in the video on demand system to minimize the hop count for finding exact media storage to retrieve video chunk data.

Mix-P2P Architecture of Distributed Storage System Based on HDFS

Aiming at the most popular Hadoop's master/slaves architecture in cloud computing, this paper presents new distributed storage architecture—MP2PDSA. Firstly, this architecture built a P2P network of SuperNodes without any master, which changed the master control of HDFS. Through the SuperNodes' collaboration, MP2PDSA enabled the storage system no longer rely on central control, which has effectively overcome Single-Point Failure, performance bottlenecks and so on. Secondly, in order to increase the availability of the whole storage system, through the coordination mechanism of P2P_SuperNodes, MP2PDSA achieved a rapid time to take over the failure subgroup without any manual intervention. Finally, MP2PDSA has a high scalability and availability compared with HDFS