Data Storage Architecture Research Articles

The evolution of data storage architectures: examining the secure value of the Data Lakehouse

The evolution of data storage architectures: examining the secure value of the Data Lakehouse

Legal Perspectives on Research Data Storage

Responsibly managing research data has become increasingly important for researchers, especially within the database research community. Despite significant progress in best practices, the state-of-the-art in research data storage is lacking from a legal perspective. We introduce the stakeholders and the dynamic nature of their relationships (such as researchers changing affiliation) and observe that no existing infrastructure for research data storage fully meets their requirements. Therefore, we emphasize the need to design a comprehensive system architecture for research data storage that is aligned with legal considerations from the start, rather than as an afterthought.

INTERFACES OF VIRTUAL DATA STORAGE IN THE CONDITIONS OF A FAST-CHANGING INFORMATION ENVIRONMENT

The article analyzes data storage architectures, explores the main approaches to cloud technologies, and highlights the features of virtual data storage architecture formation. In the process of analysis, positive and negative features of individual data storage architectures were highlighted. Cloud information technologies are considered from the point of view of their use for data storage and distribution. When considering cloud data storage, attention is focused on their architectural features from the point of view of efficiency and availability for users. Storage efficiency is an important characteristic of cloud storage infrastructure, especially given its emphasis on overall cost savings. Performance has many aspects, but the main task of a cloud storage system is to move data between the user and the remote cloud service provider. The system architecture was designed based on the proposed and modeled methods of organizing access to the cloud storage. According to the designed architecture, the data flows in the system were analyzed and investigated. The developed algorithm for the collection and integration of disparate data based on virtual data warehouses, in contrast to the Inmon and Kimball models, makes it possible to process NoSQL temporal (InfluxDB), graph (Neo4j) and document-oriented (MongoDB) data with a significantly lower value of capacity complexity and the possibility execution of recursive queries. The developed interface should be able to work with relational, temporal, graph and document-oriented data sources. To do this, it must provide the ability to perform SQL-like queries against relational databases, as well as specialized queries for each type of non-relational data. It allows to increase the efficiency of process0ng both relational and non-relational data, which makes it a powerful tool for analyzing and processing information from various sources. The development of virtual data warehouses may be associated with the introduction of new technologies, such as artificial intelligence, blockchain, quantum computing, etc. In addition, further improvements in metadata management and data security are expected.

TALICS3: Tape library cloud storage system simulator

High performance computing data is surging fast into the exabyte-scale world, where tape libraries are the main platform for long-term durable data storage besides high-cost DNA. Tape libraries are extremely hard to model, but accurate modeling is critical for system administrators to obtain valid performance estimates for their designs. This research introduces a discrete–event tape simulation platform that realistically models tape library behavior in a networked cloud environment, by incorporating real-world phenomena and effects. The platform addresses several challenges, including precise estimation of data access latency, rates of robot exchange, data collocation, deduplication/compression ratio, and attainment of durability goals through replication or erasure coding. Using the proposed simulator, one can compare the single enterprise configuration with multiple commodity library configurations, making it a useful tool for system administrators and reliability engineers. This makes the simulator a valuable tool for system administrators and reliability engineers, enabling them to acquire practical and dependable performance estimates for their enduring, cost-efficient cold data storage architecture designs.

A blockchain-based data storage architecture for Internet of Vehicles: Delay-aware consensus and data query algorithms

The recent development of the Internet of Vehicles (IoV) relies heavily on the secure store and analysis of reliable driving data. Blockchain technology has been widely used in the IoV field due to its security advantages, where consensus and query algorithms are two key modules determining its performance. However, existing solutions do not perform well enough in terms of delay performance, thus limiting their ability to support IoV environments with ultra-low-latency requirements. To address this challenge, we design a blockchain-based data storage architecture, focusing on the design of delay-aware consensus and data query algorithms. Specifically, to reduce the consensus delay, we propose a reputation-assisted and grouping-simplified Practical Byzantine Fault Tolerant (RGS-PBFT) algorithm. First, we develop a novel node credit value evaluation model by combining the behavior and configuration reputation to select the primary node with both high security and high computing capacity. Second, after determining the primary node, we reduce the communication complexity of traditional PBFT from O(Z2) to O(Z43) by establishing a grouping-simplified consensus structure. These contributions effectively reduce the consensus delay. To reduce the query delay and index construction time, we propose an index query algorithm based on the data tracking chain (DTC-Index query). By introducing the LevelDB database, it establishes a transaction index chain coupled to vehicle ID in the blockchain. This approach avoids the time-consuming block traversal during data querying. Experimental results demonstrate that the proposed algorithms outperform existing solutions, achieving significant improvements in both consensus delay and query time reduction. Notably, our solutions offer exceptional delay performance, making them well-suited for IoV environments.

Computer Vision Approach in Monitoring for Illicit and Copyrighted Objects in Digital Manufacturing

We propose a monitoring system for detecting illicit and copyrighted objects in digital manufacturing (DM). Our system is based on extracting and analyzing high-dimensional data from blueprints of three-dimensional (3D) objects. We aim to protect the legal interests of DM service providers, who may receive requests for 3D printing from external sources, such as emails or uploads. Such requests may contain blueprints of objects that are illegal, restricted, or otherwise controlled in the country of operation or protected by copyright. Without a reliable way to identify such objects, the service provider may unknowingly violate the laws and regulations and face legal consequences. Therefore, we propose a multi-layer system that automatically detects and flags such objects before the 3D printing process begins. We present efficient computer vision algorithms for object analysis and scalable system architecture for data storage and processing and explain the rationale behind the suggested system architecture.

BeeOpen—An Open Data Sharing Ecosystem for Apiculture

The digital transformation of apiculture initially encompasses Internet of Things (IoT) systems, incorporating sensor technologies to capture and transmit bee-centric data. Subsequently, data analysis assumes a vital role by establishing correlations between the collected data and the biological conditions of beehives, often leveraging artificial intelligence (AI) approaches. The field of precision bee monitoring has witnessed a surge in the collection of large volumes of diverse data, ranging from the hive weight and temperature to health status, queen bee presence, pests, and overall hive activity. Further, these datasets’ heterogeneous nature and lack of standardization present challenges in applying machine learning techniques directly to extract valuable insights. To address this issue, the envisioned ecosystem serves as an open and collaborative information platform, facilitating the exchange and utilization of bee monitoring datasets. The data storage architecture can process a large variety of data at high frequency, e.g., images, videos, audio, and time series data. The platform serves as a repository, providing crucial information about the condition of beehives, health assessments, pest attacks, swarming patterns, and other relevant data. Notably, this information portal is managed through a citizen scientist initiative. By consolidating data from various sources, including beekeepers, researchers, and monitoring systems, the platform offers a holistic view of the bee population’s status in any given area.

Hadoop-Based Political and Ideological Big Data Platform Architecture and Pattern Mining in Higher Education

There is a growing opportunity for self-improvement within the realm of higher education, represented by the combination of big data technologies and higher education. The Education Big Data Processing Framework (EBDPF) is the name of the platform that is the subject of this article, which focuses on the development of an advanced big data processing platform. In order to process huge datasets in an effective manner, the platform makes use of Hadoop's big data storage architecture, in addition to Hive, Flume, and Sqoop for data collection and synchronization, respectively. In this study, the implementation of standard data mining techniques is investigated through the use of MapReduce programming. More specifically, the study investigates the execution efficiency and scalability of these algorithms within the Hadoop platform. The data clustering task in data mining is chosen as a representative problem in this study so that the effectiveness of the EBDPF may be evaluated accordingly. An implementation of the clustering job that is based on MapReduce is created and tested on the Hadoop platform. Extensive experiments are conducted with a range of cluster sizes and data sizes. The findings provide more evidence that Hadoop distributed systems are more efficient and successful than traditional methods when it comes to managing data mining applications. The extended performance analysis of computing power not only provides evidence that the platform is currently effective, but it also provides a signal that it has a significant amount of potential that has not yet been utilized. At the convergence of big data technology and higher education, the EBDPF emerges as a strong and promising framework, demonstrating considerable steps in accelerating data processing within educational contexts. In conclusion, the EBDPF is a framework that demonstrates significant progress.

Implementation of Dynamic Virtual Cloud Architecture for Privacy Data Storage

Nowadays rapidly developing technologies, cloud computing offers versatile services. However, cloud computing presents a challenge to secure information sharing. Customers can securely share their data with others and remotely store it in the cloud using cloud storage services. In recent times, cloud storage typically represents as the primary method of external data storage. The primary challenge is safeguarding the cloud-based data against attacks. Over the information network, the growth of private or semi-private information has increased. The search techniques have not been addressed by privacy safeguards. As there is no suitable audit system, the validity of the stored data has become in question. In addition, user authentication presents additional difficulties. Hence in order to solve these issues, Design and implementation of dynamic virtual cloud architecture for privacy data storage is presented. In this approach, third-party audits are presented accompanied a new, regenerative public audit methodology. A distributed KDC (Key Distribution Center) is employed to encrypt the data. Documents can be stored on a private server in plain word form, which compromise the protection of privacy. As a result, system security can be improved to make the documents safer and more effective. The main objective of this Virtual Cloud Architecture is to achieve data confidentiality, as well as authenticity.

Smart Cart

This project is on a web-based application “Smart Cart” which ensures quality control in businesses that handle transactions revolving around consumer goods. Without proper inventory control, a large retail store may run out of stock on an important item. This system will alert the retailer when it is time to reorder. Smart Cart automates the process of managing inventory at a general store. It manages the sales, supplies and inventory of the general store. It keeps track of all the customers, the purchases made during a given period and the inventory of the items. This application is basically managed by the administrator. The administrator can login through the system. He also can view the details of the customers. A database is maintained which has the inventory of all the items. The system also keeps track of supplier details and sales. It stores all purchase order details and invoice details in respective tables in the database. This web-based application also has a POS system. The main function of a POS system is to process sales transactions quickly and accurately. When a customer makes a purchase, the system records the transaction, deducts the amount from the inventory, and calculates the total cost including taxes and discounts. Key Word: Inventory management, data storage architecture, product recommendations, invoice generation.

Data storage architecture for e-government interoperability: Morocco case

<span lang="EN-US">Nowadays, the amount of data created by the government and public sector organizations is growing at an exponential rate. Data sharing and the interoperability of e-government systems pose technological challenges. The lack of technical interoperability prevents the successful exchange and sharing of information among public organizations. To meet this challenge, enhancing interconnection and communication between different public infrastructures is an essential condition. To optimize the provisioning of storage resources, software defined storage (SDS) solutions add flexibility and adaptability to the storage process by isolating the hardware from the software. Hyper-converged infrastructure (HCI) is an emerging set of SDS solutions that provide compute, network and storage in a single platform. This paper presents a storage HCI-based architecture to store public data from different public entities, enhance collaboration and improve technical interoperability. The relevance of this approach of e-government interoperability is to allow public organization to store their data in an efficient and flexible manner on one hand, and to participate to Morocco’s e-government project on the other hand.</span>

Massive Data Storage Solution for IoT Devices Using Blockchain Technologies.

The Internet of Things (IoT) concept involves connecting devices to the internet and forming a network of objects that can collect information from the environment without human intervention. Although the IoT concept offers some advantages, it also has some issues that are associated with cyber security risks, such as the lack of detection of malicious wireless sensor network (WSN) nodes, lack of fault tolerance, weak authorization, and authentication of nodes, and the insecure management of received data from IoT devices. Considering the cybersecurity issues of IoT devices, there is an urgent need of finding new solutions that can increase the security level of WSNs. One issue that needs attention is the secure management and data storage for IoT devices. Most of the current solutions are based on systems that operate in a centralized manner, ecosystems that are easy to tamper with and provide no records regarding the traceability of the data collected from the sensors. In this paper, we propose an architecture based on blockchain technology for securing and managing data collected from IoT devices. By implementing blockchain technology, we provide a distributed data storage architecture, thus eliminating the need for a centralized network topology using blockchain advantages such as immutability, decentralization, distributivity, enhanced security, transparency, instant traceability, and increased efficiency through automation. From the obtained results, the proposed architecture ensures a high level of performance and can be used as a scalable, massive data storage solution for IoT devices using blockchain technologies. New WSN communication protocols can be easily enrolled in our data storage blockchain architecture without the need for retrofitting, as our system does not depend on any specific communication protocol and can be applied to any IoT application.

A conceptual management framework of multi-source data in R&D process

It has been 10 years since the digital thread concept was brought up. By connecting various data in different fields, digital thread is expected to solve the problem of “information silos” in the R&D process and different fields such as design and manufacturing, potentially generating an average annual output value of tens of billions of dollars for the industry. However, due to various factors such as multi-source heterogeneous data, and complex production environments, there is still no unified framework that can adapt to different scenarios. In addition, traditional data storage architectures are difficult to cope with massive amounts of data throughout the product lifecycle. To solve the problem, a conceptual management framework of multi-source data is developed based on digital thread technology, where multi-source structured data are converted into JSON data format and stored in a unified storage mode. For unstructured data, only their addresses are stored in databases. At the same time, to improve the quality of data management, the meta-database is used to store meta information related to the data. In addition, front-end and back-end data interaction and 3D model visualization are achieved based on the Django framework. The conceptual framework is demonstrated by using an illustrative case of a part design.

System Design of Integrated Intelligent Platform of National Fitness Based on K-Means Clustering.

Voice communication is the most common, direct, and effective method of information exchange between humans. Dependent speech signal processing will inevitably become an important carrier for the interaction between people and the interaction between people and computers. With the development of science and technology, data mining has become a means for users to extract effective information from a large amount of data, and many branches have been derived. Among them, K-means clustering algorithm is used as a classic clustering analysis algorithm. It is fast and simple, and it is also affected by the randomness of the initial center selection and the interference of outliers, which may cause poor clustering, but even if the above problems exist, it does not affect its wide application in various industries. This paper applies HBase storage technology and microservice framework to the fitness system and implements a national fitness system based on HBase and microservices. The system uses HBase to store fitness information, venue opening, and usage information for national fitness people; simulation results show that the accuracy rate on the data set has an obvious improvement. A fitness system that combines massive data storage and microservice architecture can improve the utilization of fitness resources, solve the problem of fitness resources, improve professional fitness levels, and provide support for the masses who regularly exercise scientifically.

Data Storage and Security in Cloud Computing: A Survey

Cloud computing is envisioned as the next generation of IT enterprise design. Cloud computing distributes application software and data bases to massive data centres, where data and service management may not be completely reliable. This creates a slew of new security issues that have yet to be properly addressed. In this article, we primarily focus on features for ensuring data storage security in the cloud, as well as architecture for data storage that is applied by other cloud service providers vendors, and essential points for establishing data storage security.

A secure neural network‐based ranking approach for document searching in cloud data center

AbstractCloud computing has gained attention due to its sophisticated processing architecture and data storage capabilities in the last couple of years. Due to high volume data and the endless number of possible users, the security of the account holder's stored data and privacy becomes essential for this paradigm. This article focuses on the encryption architecture of data storage and retrieval by creating an encrypted searchable index, which is inspired by symmetric searchable encryption. Ranking becomes a need for providing the best out of the search results to the user. This research article proposes an efficient and flexible artificial neural network (ANN) based ranking scheme to search for documents from the cloud server. The proposed algorithm architecture is segmented into three parts. The first part is the generation of the encryption index over the uploaded data, the second part is query analysis, and the third part is ranking. To consolidate the encryption mechanism, RSA, NTRU, and AES were used based on the requirement of the data. To orient the retrieval part, the degree of the top keyword in the server is determined by using term frequency with inverse document frequency schemes. The retrieved documents are further ranked using ANNs. The simulation setup was done on MATLAB 2016b having datasets from Kaggle (Twitter data) and FIRE dataset.

Trusted Data Storage Architecture for National Infrastructure.

National infrastructure is a material engineering facility that provides public services for social production and residents’ lives, and a large-scale complex device or system is used to ensure normal social and economic activities. Due to the problems of difficult data collection, long project period, complex data, poor security, difficult traceability and data intercommunication, the archives management of most national infrastructure is still in the pre-information era. To solve these problems, this paper proposes a trusted data storage architecture for national infrastructure based on blockchain. This consists of real-time collection of national infrastructure construction data through sensors and other Internet of Things devices, conversion of heterogeneous data source data into a unified format according to specific business flows, and timely storage of data in the blockchain to ensure data security and persistence. Knowledge extraction of data stored in the chain and the data of multiple regions or fields are jointly modeled through federal learning. The parameters and results are stored in the chain, and the information of each node is shared to solve the problem of data intercommunication.

Secure verifiable aggregation for blockchain-based federated averaging

IoT devices’ storage and computation capacities are constantly increasing in recent years, which brings critical challenges in data privacy protection. Federated learning (FL) and blockchain technology are two popular techniques used in IoT data aggregation, where FL enables data training with privacy protection, and blockchain provides a decentralized architecture for data storage and mining. However, very few the state-of-the-art works consider the applicability of the combination of FL and blockchain. In this paper, we adopt the federated averaging algorithm to reduce the communication overhead between the blockchain and end users to achieve higher performance. We also apply the double-mask-then-encrypt approach for end users to submit their local updates in order to protect data privacy. Finally, we propose and implement a non-interactive Public Verifiable Secret Sharing (PVSS) algorithm with Distributed Hash Table (DHT) that solves the user-drop-out problem and improves the communication efficiency between blockchain and end-users. At last, we theoretically analyze the security strengths of the proposed solution and conduct experiments to measure the execution time of PVSS on both the server and clients sides.

Decentralized Machine Autonomy for Manufacturing Servitization.

Blockchain ecosystems are rapidly maturing and meeting the needs of business environments (e.g., industry, manufacturing, and robotics). The decentralized approaches in industries enable novel business concepts, such as machine autonomy and servitization of manufacturing environments. Introducing the distributed ledger technology principles into the machine sharing and servitization economy faces several challenges, and the integration opens new interesting research questions. Our research focuses on data and event models and secure upgradeable smart contract platforms for machine servitization. Our research indicates that with the proposed approaches, we can efficiently separate on- and off-chain data and assure scalability of the DApp without compromising the trust. We demonstrate that the secure upgradeable smart contract platform, which was adapted for machine servitization, supports the business workflow and, at the same time, assures common identification and authorization of all the participants in the system, including people, devices, and legal entities. We present a hybrid decentralized application (DApp) for the servitization of 3D printing. The solution can be used for or easily adapted to other manufacturing domains. It comprises a modular, upgradeable smart contract platform and off-chain machine, customer and web management, and monitoring interfaces. We pay special attention to the data and event models during the design, which are fundamental for the hybrid data storage and DApp architecture and the responsiveness of off-chain interfaces. The smart contract platform uses a proxy contract to control the access of smart contracts and role-based access control in function calls for blockchain users. We deploy and evaluate the DApp in a consortium blockchain network for performance and privacy. All the actors in the solution, including the machines, are identified by their blockchain accounts and are compeers. Our solution thus facilitates integration with the traditional information-communication systems in terms of the hybrid architectures and security standards for smart contract design comparable to those in traditional software engineering.

Data storage architectures to accelerate chemical discovery: data accessibility for individual laboratories and the community.

As buzzwords like "big data," "machine learning," and "high-throughput" expand through chemistry, chemists need to consider more than ever their data storage, data management, and data accessibility, whether in their own laboratories or with the broader community. While it is commonplace for chemists to use spreadsheets for data storage and analysis, a move towards database architectures ensures that the data can be more readily findable, accessible, interoperable, and reusable (FAIR). However, making this move has several challenges for those with limited-to-no knowledge of computer programming and databases. This Perspective presents basics of data management using databases with a focus on chemical data. We overview database fundamentals by exploring benefits of database use, introducing terminology, and establishing database design principles. We then detail the extract, transform, and load process for database construction, which includes an overview of data parsing and database architectures, spanning Standard Query Language (SQL) and No-SQL structures. We close by cataloging overarching challenges in database design. This Perspective is accompanied by an interactive demonstration available at https://github.com/D3TaLES/databases_demo. We do all of this within the context of chemical data with the aim of equipping chemists with the knowledge and skills to store, manage, and share their data while abiding by FAIR principles.