Massive Data Storage Research Articles

Schema Extraction in NoSQL Databases: A Systematic Literature Review

Introduction: Nowadays, NoSQL databases have taken on an increasingly important role in the storage of massive data within companies. Due to a common property called schema-less, NoSQL databases offer great flexibility, particularly for the storage of data in different formats. However, despite their success in data storage, schema-less databases are a major obstacle in areas requiring precise knowledge of this schema, especially in the field of data integration. Method: This study presents a Systematic Literature Review (SLR) to explore, evaluate, and discuss relevant existing research and endeavors using novel schema extraction approaches. Furthermore, we conducted this study using a well-defined methodology to examine and study the problem of schema extraction from NoSQL databases. Results: Our research results highlight and emphasize the scheme extraction approaches and provide knowledge to researchers and practitioners by proposing schema extraction approaches and their limitations, which contributes to inventing new, more efficient approaches. Conclusion: In our future work, inspired by the recent advances in quantum computing and the emergence of post-quantum cryptography (PQC), we aim to propose a schema extraction approach that blends cutting-edge technologies with a strong focus on database security.

Massive Data HBase Storage Method for Electronic Archive Management

ABSTRACTThe acceleration of the digitalization process in enterprise and university education management has generated a massive amount of electronic archive data. In order to improve the intelligence, storage quality, and efficiency of electronic records management and achieve efficient storage and fast retrieval of data storage models, this study proposes a massive data storage model based on HBase and its retrieval optimization scheme design. In addition, HDFS is introduced to construct a two‐level storage structure and optimize values to improve the scalability and load balancing of HBase, and the retrieval efficiency of the HBase storage model is improved through SL‐TCR and BF filters. The results indicated that HDFS could automatically recover data after node, network partition, and NameNode failures. The write time of HBase was 56 s, which was 132 and 246 s less than Cassandra and CockroachDB. The query latency was reduced by 23% and 32%, and the query time was reduced by 9988.51 ms, demonstrating high reliability and efficiency. The delay of BF‐SL‐TCL was 1379.28 s after 1000 searches, which was 224.78 and 212.74 s less than SL‐TCL and Blockchain Retrieval Acceleration and reduced the delay under high search times. In summary, this storage model has obvious advantages in storing massive amounts of electronic archive data and has high security and retrieval efficiency, which provides important reference for the design of storage models for future electronic archive management. The storage model designed by the research institute has obvious advantages in storing massive electronic archive data, solving the problem of lack of scalability in electronic archive management when facing massive data, and has high security and retrieval efficiency. It has important reference for the design of storage models for future electronic archive management.

Optimization and performance improvement of distributed data storage in hybrid storage systems

With the rapid development of information technology, the storage and processing of massive data has become one of the important challenges facing the current computing field. Traditional centralized storage systems have been unable to meet the needs of big data applications, while distributed storage systems have become the infrastructure of modern data centers because of their high scalability and high availability. However, in practical applications, a single distributed storage model is often difficult to balance cost-effectiveness and performance requirements. Therefore, this paper proposes a hybrid storage system model combining local storage and cloud storage resources and discusses a series of optimization strategies for this model. The focus of the research is to improve the overall performance of the system through the design of intelligent data sharding, redundancy and fault tolerance mechanisms, and the application of effective load balancing technology. The effectiveness and superiority of the proposed method are verified by testing and analyzing several typical application scenarios on the experimental platform. The experimental results show that the optimized hybrid storage system not only significantly improves the speed of data access, but also effectively reduces the storage cost, demonstrating its potential in future large-scale data management.

Hierarchical Storage for Massive Social Network Data Based on Improved Decision Tree

Hierarchical Storage for Massive Social Network Data Based on Improved Decision Tree

Reconfigurable Multilevel Storage and Neuromorphic Computing Based on Multilayer Phase-Change Memory.

In the era of big data, the amount of global data is increasing exponentially, and the storage and processing of massive data put forward higher requirements for memory. To deal with this challenge, high-density memory and neuromorphic computing have been widely investigated. Here, a gradient-doped multilayer phase-change memory with two-level states, four-level states, and linear conductance evolution using different pulse operations is proposed. The mechanism of multilevel states is revealed through high-resolution transmission electron microscopy (HRTEM) and finite-element analysis (FEA), which show that the sequential phase change among different sublayers is realized due to the different physical properties of the sublayers with different doping concentrations. Taking advantage of the devices' linear conductance evolution characteristic, a handwritten digit (28 × 28 pixel) recognition task is implemented with a high learning accuracy of 93.46% by building a simulated artificial neural network made up of this gradient-doped multilayer phase-change memory. It is proved that this gradient-doped multilayer phase-change memory is capable of both binary multilevel digital storage and brain-inspired analog in-memory computing in the same device, enabling reconfigurable applications in the future.

Research on monitoring and energy management systems for energy storage stations on the power generation side

Abstract Addressing the challenges of limited operational efficiency, inadequate battery safety, inflexible system scalability, inadequate AGC/AVC functionality, and low levels of intelligent operation and maintenance encountered by energy storage power stations integrated with new energy generation sources, this article delves into the pivotal role of these power stations on the generation side. It meticulously examines the network architecture, functional framework, and pivotal technologies underpinning the system. Furthermore, the article proposes a comprehensive monitoring and energy management system tailored specifically for energy storage on the generation side. The system uses micro-service architecture and container technology to realize the processing and storage of massive data. Through complete isolation of the information network and control network and millisecond communication with GOOSE, the charging and discharging response speed of the energy storage station is greatly improved. The power allocation algorithm based on device status recognition completes the system’s high-precision power control of PCS, and this system uses big data analysis and data mining technology to conduct a comprehensive statistical analysis of battery health status, power station operation efficiency, and equipment operation status and energy consumption. The design of the system has favorable technical and application guidance for the subsequent energy management design and engineering application of the energy storage power station.

An Untethered Heart Rhythm Monitoring System with Automated AI‐Based Arrhythmia Detection for Closed‐Loop Experimental Application

AbstractThe heart produces bioelectrical signals, which can be measured as an electrocardiogram (ECG) for the detection of rhythm disturbances. Rapid and precise detection of these arrhythmias is crucial for their termination by closed‐looped therapeutic interventions to counteract detrimental effects. However, there is a current lack of such systems tailored for experimental cardiovascular applications. This hampers not only in‐depth mechanistic studies but also translational testing of new therapeutic strategies, especially in an untethered manner in awake animal models. To break new ground, recent advances to develop a non‐invasive AI‐supported heart rhythm monitoring system for untethered automated arrhythmia detection in a continuous manner is combined. This system is housed in a lightweight jacket for mobile use and includes an on‐skin ECG sensor, a low‐power microprocessor unit, a massive data storage unit, and a power‐management system. By implementing a novel hybrid algorithm based on so‐called heart rate (R‐R) variability and a case‐specific AI model, 100% sensitivity and 95% specificity is achieved in detecting atrial arrhythmias within 2 s upon initiation in adult rats. Thereby, the novel system sets the stage for advanced mechanistic studies and therapeutic testing, including closed‐loop applications aiming for the termination of a broad range of atrial arrhythmias.

Integrating NoSQL, Hilbert Curve, and R*-Tree to Efficiently Manage Mobile LiDAR Point Cloud Data

The widespread use of Light Detection and Ranging (LiDAR) technology has led to a surge in three-dimensional point cloud data; although, it also poses challenges in terms of data storage and indexing. Efficient storage and management of LiDAR data are prerequisites for data processing and analysis for various LiDAR-based scientific applications. Traditional relational database management systems and centralized file storage struggle to meet the storage, scaling, and specific query requirements of massive point cloud data. However, NoSQL databases, known for their scalability, speed, and cost-effectiveness, provide a viable solution. In this study, a 3D point cloud indexing strategy for mobile LiDAR point cloud data that integrates Hilbert curves, R*-trees, and B+-trees was proposed to support MongoDB-based point cloud storage and querying from the following aspects: (1) partitioning the point cloud using an adaptive space partitioning strategy to improve the I/O efficiency and ensure data locality; (2) encoding partitions using Hilbert curves to construct global indices; (3) constructing local indexes (R*-trees) for each point cloud partition so that MongoDB can natively support indexing of point cloud data; and (4) a MongoDB-oriented storage structure design based on a hierarchical indexing structure. We evaluated the efficacy of chunked point cloud data storage with MongoDB for spatial querying and found that the proposed storage strategy provides higher data encoding, index construction and retrieval speeds, and more scalable storage structures to support efficient point cloud spatial query processing compared to many mainstream point cloud indexing strategies and database systems.

AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

Effective data compression technology is essential for addressing data storage and transmission needs, especially given the escalating volume and complexity of data generated by contemporary astronomy. In this study, we propose utilizing deep learning-based lossless compression techniques to improve compression efficiency. We begin with a qualitative and quantitative analysis of the temporal and spatial redundancy in solar observation data. Based on this analysis, we introduce a novel deep learning-based framework called AstroDLLC for the lossless compression of astronomical solar images. AstroDLLC first segments high-resolution images into blocks to ensure that deep learning model training does not rely on high-computation power devices. It then addresses the non-normality of the partitioned data through simple reversible computational methods. Finally, it utilizes Bit-swap to train deep learning models that capture redundant features across multiple image frames, thereby enhancing compression efficiency. Comprehensive evaluations using data from the New Vacuum Solar Telescope reveal that AstroDLLC achieves a maximum compression ratio of 3.00 per image, surpassing Gzip, RICE, and other lossless technologies. The performance of AstroDLLC underscores its potential to address data compression challenges in astronomy.

Pairing-free Proxy Re-Encryption scheme with Equality Test for data security of IoT

The construction of IoT cloud platform brings great convenience to the storage of massive IoT node data. To ensure security of data collected by the IoT nodes, the data must be encrypted when the nodes upload it. However, it arises some challenge problems, such as efficient retrieval on encrypted data, secure de-duplication and massive data recovery. Proxy Re-Encryption with Equality Test (PREET) can be used to solve these problems. But, existing PREET schemes are based on bilinear pairings, which have low operational efficiency. In order to improve the overall operational efficiency, this paper constructs a Pairing-Free PREET scheme (PF-PREET). Security of this scheme is guaranteed by the Gap Computational Diffie–Hellman assumption in random oracle model. It is demonstrated that the PF-PREET scheme can achieve ciphertext indistinguishability for malicious users and one-wayness for malicious servers. It still has the same security level compared to existing PREET schemes by expanding the scope of the equality test. We discussed the efficiency of the proposed PF-PREET scheme in terms of overall running time, average running time of each phase and time overhead of tag test. The simulated experiment shows a certain improvement compared to PREET schemes using bilinear pairing.

Caching or not: An online cost optimization algorithm for geodistributed data analysis in cloud environments

With the wide application of big data technology, a large number of data geographically stored in data centers across various regions are generated everyday, waiting to be analyzed by big data tasks. Examples of such data analysis tasks include weather prediction and intelligent healthcare applications. Clouds are being used by more and more enterprises due to their nearly infinite resources, ease of scaling, and other characteristics. Organizations often rent StaaS (Storage-as-a-Service) storage products offered by cloud providers, such as OSS (Object Storage Service), for massive data storage, while building a big data cluster in cloud environments to analyze the collected data from various regions. However, when the data to be analyzed are not located in the rented cluster, how to efficiently and economically process the distributed input data stored in clouds becomes an urgent problem to be solved. A simple approach is to only cache frequently accessed data rather than all data from other regions into the cluster to reduce total traffic costs. However, it is generally very hard to predict future data access curves. Thus, a rash caching decision may incur more costs. To address this problem, in this paper we propose an online algorithm for guiding cloud users to make cost-effective caching decisions properly, while not requiring any future information. We prove theoretically that the competitive ratio of our online algorithm is less than 2. Finally we verify the effectiveness of our proposed algorithm through extensive experiments based on the real price of Alibaba’s public IaaS cloud products using both real-world Yahoo S2 data and synthesized datasets.

Management information platform based on Internet of Things

In order to study the financial management mode and innovation of electric power enterprises under the current situation, a research method of electric power financial management innovation in the era of Internet of Things is proposed. According to the network topology of the system and the analysis of the Android system, the client of the financial management system, the main functional modules of the system and the data warehouse are designed. The client of the financial management system will divide the client into an information interaction unit and a data calculation unit according to the actual needs of the client; the main functional modules of the system are divided into account management unit, salary management unit, project management unit, system management unit, and fixed asset management unit. Unit, revenue and expenditure management unit and report management unit, the sub-units of each unit are divided into brief analysis, and the outpatient registration fee management module and inpatient fee management module of the revenue and expenditure management unit are introduced in detail; in the hospital financial data warehouse, in the application layer An interactive platform is built between it and the data warehouse, and the module is integrated into the existing data warehouse to realize the processing, storage and operation of massive data. The proposed system has a large operating safety factor, high user satisfaction, and an average system flexibility factor of 0.94. The power financial management information platform based on the Internet of Things has a very good performance in the application. Therefore, the method in this paper can enhance the work efficiency of electric power enterprises and enhance the core competitiveness of electric power enterprises.

Non-Uniform Spatial Partitions and Optimized Trajectory Segments for Storage and Indexing of Massive GPS Trajectory Data

The presence of abundant spatio-temporal information based on the location of mobile objects in publicly accessible GPS mobile devices makes it crucial to collect, analyze, and mine such information. Therefore, it is necessary to index a large volume of trajectory data to facilitate efficient trajectory retrieval and access. It is difficult for existing indexing methods that primarily rely on data-driven indexing structures (such as R-Tree) or space-driven indexing structures (such as Quadtree) to support efficient analysis and computation of data based on spatio-temporal range queries as a service basis, especially when applied to massive trajectory data. In this study, we propose a massive GPS data storage and indexing method based on uneven spatial segmentation and trajectory optimization segmentation. Primarily, the method divides GPS trajectories in a large spatio-temporal data space into multiple MBR sequences by greedy algorithm. Then, a hybrid indexing model for segmented trajectories is constructed to form a global spatio-temporal segmentation scheme, called HHBITS index, to achieve hierarchical organization of trajectory data. Eventually, a spatio-temporal range query processing method is proposed based on this index. This paper implements and evaluates the index in MongoDB and compares it with two other spatio-temporal composite indexes for performing spatio-temporal range queries efficiently. The experimental results show that the method in this paper has high performance in responding to spatio-temporal queries on large-scale trajectory data.

Privacy-Preserving Graph Matching Query Supporting Quick Subgraph Extraction

Graph matching, as one of the most fundamental problems in graph database, has a wide range of applications. Due to the large scale of graph database and the hardness of graph matching, it is an attractive alternative to make use of the cloud to store massive data graphs and conduct the complex graph matching. To protect the privacy, data graphs are usually encrypted before being outsourced to the cloud. A few schemes have been proposed to support graph matching query over encrypted graphs. However, none of them can realize efficient subgraph extraction when the matched subgraph needs to be exactly located at the data graph. The graph user has to perform the complex subgraph isomorphism operation to extract the isomorphic subgraph from the matched data graph in state-of-the-art schemes. The time complexity of judging subgraph isomorphism is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(n!n^{2})$</tex-math></inline-formula> at most, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> is the number of vertices in the data graph. In order to solve this problem, we propose a privacy-preserving graph matching query scheme supporting quick subgraph extraction in this paper. In our design, two non-colluding cloud servers are adopted to accomplish the matching operation jointly. Neither of them can infer the plaintexts of graphs. The first cloud server can prune some impossible matched vertices from the data graph, and get a possible matched matrix to represent which vertices in the data graph might match with the vertices in the query graph. The second cloud server performs the related matrix operations based on its stored information and the information sent by the first cloud server. The two cloud servers jointly verify the correctness of the matched matrix. The graph user can directly and quickly extract the matrix of the subgraph isomorphic to the query graph from the data graph matrix based on the matched matrix, and further recover this subgraph. No subgraph isomorphism operation is involved in the procedure of subgraph extraction for the graph user. The time complexity of subgraph extraction is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(m^{2})$</tex-math></inline-formula> in our scheme, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$m$</tex-math></inline-formula> is the number of vertices in the query graph. The extensive experiments with real-world database demonstrate the efficiency of the proposed privacy-preserving graph matching scheme.

Home Automation Using AI and IoT

The IoT network's massive data flows and storage can be easily managed with AI. IoT is currently growing more and more popular because to the development of high-speed internet networks and several cutting-edge sensors that can be incorporated into microcontrollers. The data that is now flowing across the internet includes user data that is sent and received from workstations as well as sensor data. Fewer data may really be experiencing challenges on the storage side, latency, channels limitation, and suffocation inside the networks due to the increase in the number of workstations and numerous sensors. Over the past ten years, several algorithms have been presented to prevent these issues. AI is still the only algorithm that can manage and regulate network congestion in addition to mining information. This paper's purpose is to demonstrate how an AI system is being used in the Internet of Things. The study emphasizes the significance of information management and mining. Additionally, the approach used in synthetic intelligence, such as fuzzy logics or neural networks, will be examined in this study in relation to IoT network. The software in specified networks and self-modifying networks are some of the essential parameters in the AI IoT system.

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments.

With the development of synchrotron radiation sources and high-frame-rate detectors, the amount of experimental data collected at synchrotron radiation beamlines has increased exponentially. As a result, data processing for synchrotron radiation experiments has entered the era of big data. It is becoming increasingly important for beamlines to have the capability to process large-scale data in parallel to keep up with the rapid growth of data. Currently, there is no set of data processing solutions based on the big data technology framework for beamlines. Apache Hadoop is a widely used distributed system architecture for solving the problem of massive data storage and computation. This paper presents a set of distributed data processing schemes for beamlines with experimental data using Hadoop. The Hadoop Distributed File System is utilized as the distributed file storage system, and Hadoop YARN serves as the resource scheduler for the distributed computing cluster. A distributed data processing pipeline that can carry out massively parallel computation is designed and developed using Hadoop Spark. The entire data processing platform adopts a distributed microservice architecture, which makes the system easy to expand, reduces module coupling and improves reliability.

Analysis of the Application of Storage Resource Pool Technology of Inner Mongolia Power based on the Background of Big Data

With the continuous deepening of the Inner Mongolia Power Company's level of informatization and the continuous expansion of the scale of business and application system construction, the company will also enter the era of big data, and the amount of data will show an explosive growth trend. In order to solve the problems of massive data storage and data value mining faced by the enterprise during its development process, research on the application of storage resource pool technology has been conducted. Ultimately, the unified planning, allocation, and management of storage resources have been achieved, greatly improving the utilization efficiency of storage resources.

The new analytical and numerical analysis of 2D stretching plates in the presence of a magnetic field and dependent viscosity

This study explores heat transfer in a system involving Jeffery fluid of MHD flow and a porous stretching sheet. The mathematical representation of this system is initially described using a partial differential equation (PDE), which is then converted into an ordinary differential equation (ODE) through numerical techniques such as Lie similarity and transformation methods, along with the shooting approach. The results indicate that altering the variables of Jeffery fluid, heat source, porosity on a stretching sheet, and the physical characteristics of the magnetic field within the system leads to an upward trend. Implementing this enhanced heat transfer system can yield benefits across various domains, including industrial machinery, mass data storage units, electronic device cooling, etc., thereby enhancing heating and cooling processes. Furthermore, the study also utilized Akbari-Ganji’s Method, a new semi-analytical method designed to solve nonlinear differential equations of heat and mass transfer. The results obtained from this method were compared with those from the finite element method for accuracy, efficiency, and simplicity. This research provides valuable insights into heat transfer dynamics in complex systems and offers potential applications in various industrial settings. It also contributes to developing more efficient and effective heat transfer techniques.

SSTU: Swin-Spectral Transformer U-Net for hyperspectral whole slide image reconstruction

Whole Slide Imaging and Hyperspectral Microscopic Imaging provide great quality data with high spatial and spectral resolution for histopathology. Existing Hyperspectral Whole Slide Imaging systems combine the advantages of the techniques above, thus providing rich information for pathological diagnosis. However, it cannot avoid the problems of slow acquisition speed and mass data storage demand. Inspired by the spectral reconstruction task in computer vision and remote sensing, the Swin-Spectral Transformer U-Net (SSTU) has been developed to reconstruct Hyperspectral Whole Slide images (HWSis) from multiple Hyperspectral Microscopic images (HMis) of small Field of View and Whole Slide images (WSis). The Swin-Spectral Transformer (SST) module in SSTU takes full advantage of Transformer in extracting global attention. Firstly, Swin Transformer is exploited in space domain, which overcomes the high computation cost in Vision Transformer structures, while it maintains the spatial features extracted from WSis. Furthermore, Spectral Transformer is exploited to collect the long-range spectral features in HMis. Combined with the multi-scale encoder-bottleneck-decoder structure of U-Net, SSTU network is formed by sequential and symmetric residual connections of SSTs, which reconstructs a selected area of HWSi from coarse to fine. Qualitative and quantitative experiments prove the performance of SSTU in HWSi reconstruction task superior to other state-of-the-art spectral reconstruction methods.

Big data Research on the Application of Tracking Policy Audit in the Implementation of Tax Reduction and Fee

Since 2018, chinese government have conducted tracking audits on the implementation of tax reduction and fee reduction policies, facing the challenges of insufficient audit resources and massive unstructured data storage. This article attempts to introduce big data technology into tracking audit policies. This article elaborates on the importance and urgency of tracking and auditing the implementation of tax and fee reduction policies. The impact of big data technology in the field of auditing on the tracking and auditing of the implementation of tax and fee reduction policies is also discussed. It is proposed that auditing agencies should apply big data technology to the tracking and auditing of the implementation of tax and fee reduction policies. And we also attempted to design an audit plan based on Neo4j graphical database technology, which includes audit objectives and scope, audit content and focus, specific big data audit steps and audit results. In response to the problems that hinder the application of big data technology, suggestions have been put forward to improve the standardization of the data process system, actively promote more efficient big data technology, and increase the training of big data audit talents.