Efficient Data Storage Research Articles

Quantum 3.0: Quantum Learning, Quantum Heuristics and Beyond

Quantum learning paradigms address the question of how best to harness conceptual elements of quantum mechanics and information processing to improve operability and functionality of a computing system for specific tasks through experience. It is one of the fastest evolving framework, which lies at the intersection of physics, statistics and information processing, and is the next frontier for data sciences, machine learning and artificial intelligence. Progress in quantum learning paradigms is driven by multiple factors: need for more efficient data storage and computational speed, development of novel algorithms as well as structural resonances between specific physical systems and learning architectures. Given the demand for better computation methods for data-intensive processes in areas such as advanced scientific analysis and commerce as well as for facilitating more data-driven decision-making in education, energy, marketing, pharmaceuticals and healthcare, finance and industry.

Circuit-Level Realization of an Adaptive Trending Prediction Algorithm for Lossless ECG Compression

This paper introduces an advanced approach to the hardware implementation of lossless compression for electrocardiogram (ECG) signals, combining Content Adaptive Golomb Rice (CAGR) coding with Adaptive Trending Prediction (ATP). The CAGR technique dynamically adjusts its parameters based on local signal characteristics, effectively optimizing compression efficiency. Simultaneously, Adaptive Trending Prediction analyzes and predicts signal trends to further minimize redundancy before encoding, enhancing overall compression ratios. The hardware architecture presented in this study is meticulously designed to leverage these techniques, ensuring efficient utilization of resources while maintaining real time performance suitable for medical applications. Design considerations include algorithmic complexity, throughput, and resource utilization metrics tailored to meet the stringent requirements of medical monitoring systems. Experimental evaluations validate the efficacy of the proposed method, demonstrating substantial reductions in data size while preserving clinical fidelity crucial for accurate diagnosis and monitoring. The findings underscore the feasibility and practicality of integrating advanced compression algorithms into dedicated hardware platforms, thereby facilitating more efficient data transmission and storage in medical environments.

Geomatics-enabled Interdisciplinary Approach Based on Geospatial Data Processing for Hydrogeological Risk-analysis

Hydrogeological risks that are associated with rivers have emerged as a significant concern worldwide, impacting both natural ecosystems and human settlements. This contribution presents an interdisciplinary project that leverages many technologies for data-acquisition and modeling to comprehensively analyze and manage risks in riverine environments. The project integrates geomatics, geological, and hydrological techniques to provide a holistic understanding of river dynamics and their associated hazards. As a central component of this project, geomatics plays a pivotal role in instrumental field surveying through the deployment of photogrammetry and LiDAR instruments. Remote-sensing data from satellite imagery further enriches the project’s temporal analysis capabilities. By analyzing this data over time, researchers can monitor changes in river patterns, land use, and climate-related variables; this helps identify trends and potential triggers for hydrological events. To manage and integrate the vast amount of geospatial information that is generated, a geodatabase within a geographic information system (GIS) has been established. It enables efficient data storage, retrieval, and analysis, fostering collaboration among multidisciplinary researcher teams. This system offers tools for risk-assessment, modeling, and scenario planning; these allow for proactive measures for mitigating hydrological risks.

Data Acquisition, Processing, and Aggregation in a Low-Cost IoT System for Indoor Environmental Quality Monitoring

The rapid spread of Internet of Things technologies has enabled a continuous monitoring of indoor environmental quality in office environments by integrating monitoring devices equipped with low-cost sensors and cloud platforms for data storage and visualization. Critical aspects in the development of such monitoring systems are effective data acquisition, processing, and visualization strategies, which significantly influence the performance of the system both at monitoring device and at cloud platform level. This paper proposes novel strategies to address the challenges in the design of a complete monitoring system for indoor environmental quality. By adopting the proposed solution, one can reduce the data rate transfer between the monitoring devices and the server without loss of information, as well as achieve efficient data storage and aggregation on the server side to minimize retrieval times. Finally, enhanced flexibility in the dashboard for data visualization is obtained, thus enabling graph modifications without extensive coding efforts. The functionality of the developed system was assessed, with the collected data in good agreement with those from other instruments used as references.

Asynchronous verifiable information dispersal protocol of achieving optimal communication

The evolution of distributed systems necessitates robust protocols to facilitate secure and efficient data storage and retrieval amid challenges like asynchronous communication and adversarial threats. However, achieving the theoretical lower bounds of communication cost in Asynchronous Verifiable Information Dispersal (AVID) protocols remains an unresolved issue, prompting the need for innovation in this domain.In this paper, we propose a new AVID protocol designed to optimize communication cost. In contrast to previous approaches, we leverage vector commitment (VC) technology combined with erasure codes, replacing conventional hash tree structures or error correction codes. This approach results in significant O(logn) communication savings. Additionally, our research reveals that while ensuring simultaneous output from all honest nodes is desirable to satisfy Totality, it is not necessary for all nodes to receive corresponding shares or proofs during the Dispersal phase. This insight allows us to circumvent the common O(n2κ) communication overhead.Our AVID protocol achieves an optimal dispersal cost of O(|M|+n2) compared to the prior best of O(|M|+κn2). Furthermore, each node in our protocol incurs a storage cost of O(|M|/n+κ) bits, while the retrieval cost per client is O(|M|+nκ).

Research on Edge Cloud Data Storage Method of Power Operation Site in Internet of Things Environment Based on Paxos Algorithm

Abstract In order to realize the efficient storage of electric power job site data and ensure the utilization effect of the stored data, a method of electric power job site edge cloud data storage based on the Paxos algorithm is proposed. In this method, power field operation data are collected comprehensively through the edge server and edge controller in the edge computing module, the collected data are processed, and the data storage tasks are allocated reasonably. The allocated data are transferred to the data storage module, which identifies the bad data in the data through a gap statistics algorithm and cluster analysis and retains the valid and normal data. After the primary and secondary nodes are determined based on the Paxos algorithm, the data storage model of power operation site is constructed. After the consistency detection of the reserved data, the data storage of power operation site is completed. The platform management module can analyze the stored data and present the analysis results to the application decision-making module for presentation. The test results show that the maximum time delay and energy consumption are 2.15 s and 42.6 W, respectively, when the method is used for data transmission, which can accurately identify the bad data in the field data of electric power operation, and the reliability index results of data consistency detection are all above 0.95, which ensures the good consistency of the stored power operation site data and effectively completes the power operation site data storage.

Multimodal Environmental Sensing Using AI & IoT Solutions: A Cognitive Sound Analysis Perspective.

This study presents a novel audio compression technique, tailored for environmental monitoring within multi-modal data processing pipelines. Considering the crucial role that audio data play in environmental evaluations, particularly in contexts with extreme resource limitations, our strategy substantially decreases bit rates to facilitate efficient data transfer and storage. This is accomplished without undermining the accuracy necessary for trustworthy air pollution analysis while simultaneously minimizing processing expenses. More specifically, our approach fuses a Deep-Learning-based model, optimized for edge devices, along with a conventional coding schema for audio compression. Once transmitted to the cloud, the compressed data undergo a decoding process, leveraging vast cloud computing resources for accurate reconstruction and classification. The experimental results indicate that our approach leads to a relatively minor decrease in accuracy, even at notably low bit rates, and demonstrates strong robustness in identifying data from labels not included in our training dataset.

Intelligent Edge-powered Data Reduction: A Systematic Literature Review

The development of the Internet of Things (IoT) paradigm and its significant spread as an affordable data source has brought many challenges when pursuing efficient data collection, distribution, and storage. Since such hierarchical logical architecture can be inefficient and costly in many cases, Data Reduction (DR) solutions have arisen to allow data preprocessing before actual transmission. To increase DR performance, researchers are using Artificial Intelligence (AI) techniques and models toward reducing sensed data volume. AI for DR on the edge is investigated in this study in the form of a Systematic Literature Review (SLR) encompassing major issues such as data heterogeneity and AI-based techniques to reduce data, architectures, and contexts of usage. An SLR is conducted to map the state of the art in this area, highlighting the most common challenges and potential research trends in addition to a proposed taxonomy.

Efficient Data Storage Management and Reducing Deduplication in Cloud Computing

Efficient Data Storage Management and Reducing Deduplication in Cloud Computing

Efficient DNA-based data storage using shortmer combinatorial encoding

Data storage in DNA has recently emerged as a promising archival solution, offering space-efficient and long-lasting digital storage solutions. Recent studies suggest leveraging the inherent redundancy of synthesis and sequencing technologies by using composite DNA alphabets. A major challenge of this approach involves the noisy inference process, obstructing large composite alphabets. This paper introduces a novel approach for DNA-based data storage, offering, in some implementations, a 6.5-fold increase in logical density over standard DNA-based storage systems, with near-zero reconstruction error. Combinatorial DNA encoding uses a set of clearly distinguishable DNA shortmers to construct large combinatorial alphabets, where each letter consists of a subset of shortmers. We formally define various combinatorial encoding schemes and investigate their theoretical properties. These include information density and reconstruction probabilities, as well as required synthesis and sequencing multiplicities. We then propose an end-to-end design for a combinatorial DNA-based data storage system, including encoding schemes, two-dimensional (2D) error correction codes, and reconstruction algorithms, under different error regimes. We performed simulations and show, for example, that the use of 2D Reed-Solomon error correction has significantly improved reconstruction rates. We validated our approach by constructing two combinatorial sequences using Gibson assembly, imitating a 4-cycle combinatorial synthesis process. We confirmed the successful reconstruction, and established the robustness of our approach for different error types. Subsampling experiments supported the important role of sampling rate and its effect on the overall performance. Our work demonstrates the potential of combinatorial shortmer encoding for DNA-based data storage and describes some theoretical research questions and technical challenges. Combining combinatorial principles with error-correcting strategies, and investing in the development of DNA synthesis technologies that efficiently support combinatorial synthesis, can pave the way to efficient, error-resilient DNA-based storage solutions.

A secure framework for effective workload resource management

An efficient and dynamic role-based access-control (RBAC) model is presented in this work which utilizes access-control for internet of things (IoT) nodes while minimizing storage and computational overhead. Also, for the identification of the malicious packets at the gateway server, a machine learning method has been presented. In addition, a framework for data management techniques in the IoT environment is designed to ensure efficient and secure storage, management, and processing of IoT data. The results have been evaluated by using the Montage and Cybershake workload in terms of energy consumption, processing time, detection accuracy and misclassification rate. The results show that the proposed secure framework for effective workload resource management (SFE-WRM) attains better performance in comparison to the reliable and energy‐efficient route selection (REERS) and FTA-WRM method. Also, by using the security method, the proposed method provides better security to the IoT nodes during the data aggregation and processing of the workload. The ultimate aim of this work is to provide a solution for the development of a secure and efficient IoT environment that can address critical security challenges and enable the widespread adoption of IoT devices and services.

Integrating Azure Services for Real Time Data Analytics and Big Data Processing

Integrating Azure services for real-time data analytics and big data processing is a transformative approach that leverages the power of cloud computing to handle vast amounts of data at scale. Azure offers a wide array of services such as Azure Synapse Analytics, Azure Data Lake, Azure Stream Analytics, and Azure Databricks, which together enable businesses to extract valuable insights from both structured and unstructured data in real time. These services provide seamless integration, allowing for real-time data ingestion, transformation, and analysis, facilitating quicker decision-making processes. Azure’s capabilities in big data processing allow for scalable and efficient storage of data, ensuring optimal performance even as data volumes grow. By utilizing tools like Azure Data Factory and Azure HDInsight, organizations can automate complex workflows and apply machine learning algorithms for predictive analytics, driving innovation and competitive advantage. The integration of Azure's real-time analytics services also supports the use of advanced technologies like artificial intelligence (AI) and Internet of Things (IoT), enabling companies to monitor live streams of data from various devices and platforms. This infrastructure not only enhances operational efficiency but also ensures a more proactive approach to problem-solving by identifying trends and anomalies as they occur. In conclusion, Azure’s comprehensive suite of services offers a robust platform for managing and analysing big data in real time, empowering organizations to leverage data for better decision-making and improved business outcomes.

HealthCare education Platform

Our health education platform is designed to provide reliable health information and resources in a user-friendly manner. It features a responsive web design using modern technologies for seamless browsing. The backend utilizes Django or Express.js for robust logic and database handling, with PostgreSQL or MongoDB for efficient data storage. Elasticsearch or Solr powers fast and accurate search functionality. Secure authentication and authorization are ensured through JWT or OAuth. Comments and feedback are managed with moderation tools and notifications. Performance is optimized with monitoring tools, caching, and CDN integration. Scalability is achieved through cloud infrastructure and load balancing. Comprehensive security measures include HTTPS encryption and regular audits. The platform allows users to explore a wide range of health-related content, including articles, blogs, and expert opinions, with personalized content suggestions. By offering a comprehensive and user-friendly platform, we aim to empower individuals to make informed decisions about their health and well-being.

Design and Development of Model to Detect Garbage from CCTV

Urban waste management is a pressing concern globally, necessitating innovative solutions for effective sanitation and sustainability. In response, this project introduces a Garbage Detection System leveraging deep learning techniques and web-based monitoring for real-time surveillance of city-wide Garbage accumulation. Utilizing the YOLOv7 object detection model, the system achieves an accuracy rate of 81% in identifying Garbage objects across diverse environmental conditions and camera angles. The accompanying web application dashboard provides intuitive access to live camera feeds, empowering stakeholders with proactive monitoring capabilities and facilitating prompt response to sanitation issues. Integration with MongoDB ensures efficient data storage and retrieval, enabling seamless access to historical Garbage detection data for analysis and optimization. The iterative refinement process, guided by periodic model updates and user feedback loops, contributes to continuous improvement in system performance. Through its comprehensive approach, this project demonstrates the feasibility and effectiveness of utilizing advanced computer vision technology for enhancing urban cleanliness and sustainability. Future endeavours will focus on further enhancing accuracy, scalability, and integration with existing waste management infrastructure to realize holistic urban waste management solutions. Key Words: YOLOv7, neural network, object detection, deep learning, web-based monitoring

Efficient Storage of Heterogeneous IoT data in a Blockchain using an Indexing Method in Metric Space

In this work, we proposed a IoT data indexing method to surpass some challenges encountered during the use of hashing in the storage of data in a blockchain. The indexing method was developed in metric space in which no dimensions are considered and only distance between objects is taken into account. The proposed method consisted on putting the index in the inner of a block. The index, called GHB-tree is based on space partitioning using hyperplane. The proposed approach was tested using two datasets of close size and different dimensions. The experimental results showed that the proposed method is efficient and competitive to other storing methods since the queries retrieve time is very reduced to be expressed by millisecond compared with that of other blockchains.

Fish Tank: An IoT-Based Mini Aquarium Control System

The manual management of aquarium systems is a meticulous process involving hands-on monitoring of various critical parameters such as water quality, feeding, lighting, and maintenance. However, this approach comes with challenges, including significant time consumption, the potential for human errors, and a lack of remote monitoring capabilities. To overcome these challenges, the development of automated smart aquarium systems has emerged as a transformative solution. These systems aim to simplify maintenance, enhance the well-being of aquatic life, provide remote accessibility, and contribute valuable data for research and education in the field. In the context of this evolutionary shift, this paper proposes a comprehensive Fish Tank system built upon the innovative IoT solution, IoT talk. This system utilizes sensors to drive actuators in real-time, enabling intelligent control of various water conditions within the aquarium. A notable feature of this proposed system is the implementation of a precise fish-feeding mechanism, enhancing the overall care provided to aquatic life. The paper goes further to present an analytic model, simulation, and measurement experiments that delve into the effects of IoT message delays and loss on water condition control, contributing valuable insights to the efficiency of the proposed system. Moreover, the proposed Fish Tank system incorporates advanced monitoring tools seamlessly connected to a robust database. This integration allows for efficient data storage and retrieval, enhancing the overall performance and reliability of the system. By extending this connectivity to a mobile application, the solution not only reduces human effort and errors in aquarium management but also provides users with convenient remote access. This transformative approach ensures continuous monitoring and control of the aquarium environment, ultimately contributing to the sustained health and well-being of aquatic life

An Efficient NoSQL-Based Storage Schema for Large-Scale Time Series Data

In IoT (internet of things), most data from the connected devices change with time and have sampling intervals, which are called time-series data. It is challenging to design a time series storage model that can write massive time-series data in a short time and can query and analyze the persistent time-series data for a long time. This paper constructs the RHTSDB (Redis-HBase Time Series Database) storage model based on Redis and HBase. RHTSDB uses the memory database Redis (Remote Dictionary Server) to cache massive time-series data, providing efficient data storage and query functions. HBase is used in RHTSDB for long-term storage of time-series data to realize their persistence. The paper designs a cold and hot separation mechanism for time-series data, where the infrequently accessed cold data are stored in HBase, and the frequently accessed and latest data are stored in Redis. Experiments verify that RHTSDB has apparent advantages over Apache IoTDB and HBase in data intake and query efficiency.

Assessing Riverbank Change Caused by Sand Mining and Waste Disposal Using Web-Based Volunteered Geographic Information

River water is one of the most important natural resources for economic development and environmental sustainability. However, river water systems are vulnerable in some of the densely populated regions across the globe. Intense sand mining and waste disposal can lead to river changes, loose foundations, and biodiversity loss. This study aims to develop a web-based geographic information system (GIS) platform to monitor river channel changes and their impacts on river environments due to sand mining and waste burial. The system integrates open-source software, Windows Server 2012, a web server, and PostgreSQL with PostGIS plugins for efficient mapping and storage of geospatial data and volunteered reporting of location events. Interferometric methods, including SNAP2STAMPS Automated Algorithm, persistent scatterer interferometry (PSI), small baseline subset (SBAS), and Snap software, were used to analyze spatial changes of subsidence from Sentinel-1 satellite data from 2021 to 2023 in the Gadar River in Oshnavieh, Iran. The results showed that the maximum subsidence at the riverbank was −10.1 cm due to sand mining, and the maximum uplift was 8.2 cm due to waste landfilling. The average subsidence was reported to be −5.1 cm. The results emphasize spatial analysis, showcasing material mining’s impact on subsidence trends and underscoring the significance of public participation in monitoring river health. Three parameters—completeness, correctness, and quality—were used to validate the system. Validation results showed completeness, correctness, and quality rates of 94.15%, 92.48%, and 86.63%, respectively.

NeuralVDB: High-resolution Sparse Volume Representation using Hierarchical Neural Networks

We introduce NeuralVDB, which improves on an existing industry standard for efficient storage of sparse volumetric data, denoted VDB [Museth 2013 ], by leveraging recent advancements in machine learning. Our novel hybrid data structure can reduce the memory footprints of VDB volumes by orders of magnitude, while maintaining its flexibility and only incurring small (user-controlled) compression errors. Specifically, NeuralVDB replaces the lower nodes of a shallow and wide VDB tree structure with multiple hierarchical neural networks that separately encode topology and value information by means of neural classifiers and regressors respectively. This approach is proven to maximize the compression ratio while maintaining the spatial adaptivity offered by the higher-level VDB data structure. For sparse signed distance fields and density volumes, we have observed compression ratios on the order of 10× to more than 100× from already compressed VDB inputs, with little to no visual artifacts. Furthermore, NeuralVDB is shown to offer more effective compression performance compared to other neural representations such as Neural Geometric Level of Detail [Takikawa et al. 2021 ], Variable Bitrate Neural Fields [Takikawa et al. 2022a ], and Instant Neural Graphics Primitives [Müller et al. 2022 ]. Finally, we demonstrate how warm-starting from previous frames can accelerate training, i.e., compression, of animated volumes as well as improve temporal coherency of model inference, i.e., decompression.

RANSAC Algorithm and Distributed Framework for Point Cloud Processing of Ancient Buildings

This paper introduces a comprehensive framework for the point cloud processing of traditional buildings. The framework includes segmentation using the RANSAC algorithm and distributed storage based on a fuzzy weighting approach. The methodology employs a height threshold parameter for segmentation to extract building structural elements effectively. Furthermore, an interactive access control model distributes tasks across nodes to achieve load balancing during point cloud matching and analysis. Experimental results demonstrate segmentation accuracy exceeding 99% and alignment time reduction to 967 ms for point cloud models. The distributed computation efficiency reaches 0.8, outperforming conventional methods. The proposed techniques enable accurate dimensional capture, efficient data storage, and information extraction from traditional buildings to support digital preservation.