Data Management Solutions Research Articles

Modular approach to near-time data management for multi-city atmospheric environmental observation campaigns

Abstract. Urban observation networks are becoming denser, more diverse, and more mobile, while being required to provide results in near time. The Synergy Grant “urbisphere” funded by the European Research Council (ERC) has multiple simultaneous field campaigns in cities of different sizes, collecting data to improve weather and climate models and services, including assessing the impact of cities on the atmosphere (e.g., heat, moisture, pollutant, and aerosol emissions) and people's exposure to extremes (e.g., heat waves, heavy precipitation, air pollution episodes). Here, a solution to this challenge for facilitating diverse data streams from multiple sources, scales (e.g., indoors, regional-scale atmospheric boundary layer), and cities is presented. For model development and evaluation in heterogeneous urban environments, we need meshed networks of in situ observations with ground-based and airborne (remote) sensing platforms. In this contribution we describe challenges, approaches, and solutions for data management, data infrastructure, and data governance to handle the variety of data streams from primarily novel modular observation networks deployed in multiple cities, in combination with existing data collected by partners, ranging in scale from indoor sensor deployments to regional-scale boundary layer observations. A metadata system documents (1) sensors and instruments, (2) the location and configuration of deployed components, and (3) maintenance and events. This metadata system provides the backbone for converting instrument records to calibrated, location-aware, convention-aligned, and quality-assured data products, according to FAIR (findable, accessible, interoperable, and reusable) principles. The data management infrastructure provides services (via, e.g., Application Programming Interface – APIs, apps, integrated computing interfaces – ICEs) for data inspection and subsequent calculations by campaign participants. Some near-real-time distributions are made to international networks (e.g., AERONET, PhenoCam) or local agencies (e.g., GovDATA) with appropriate attribution. The data documentation conventions, used to ensure structured datasets, in this case are used to improve the delivery of integrated urban services, such as to research and operational agencies, across many cities.

Flexible and Effective Object Tiering for Heterogeneous Memory Systems

Computing platforms that package multiple types of memory, each with their own performance characteristics, are quickly becoming mainstream. To operate efficiently, heterogeneous memory architectures require new data management solutions that are able to match the needs of each application with an appropriate type of memory. As the primary generators of memory usage, applications create a great deal of information that can be useful for guiding memory management, but the community still lacks tools to collect, organize, and leverage this information effectively. To address this gap, this work introduces a novel software framework that collects and analyzes object-level information to guide memory tiering. The framework includes tools to monitor the capacity and usage of individual data objects, routines that aggregate and convert this information into tier recommendations for the host platform, and mechanisms to enforce these recommendations according to user-selected policies. Moreover, the developed tools and techniques are fully automatic, work on standard Linux systems, and do not require modification or recompilation of existing software. Using this framework, this study evaluates and compares the impact of a variety of design choices for memory tiering, including different policies for prioritizes objects for fast memory tier as well as the frequency and timing of migration events. The results, collected on a modern Intel ® platform with conventional DDR4 SDRAM as well as Intel Optane NVRAM, show that guiding data tiering with object-level information can enable significant performance and efficiency benefits compared to standard hardware- and software-directed data tiering strategies for a diverse set of memory-intensive workloads.

Optimizing Linked List-based Smart Contract on Ethereum with IPFS for E-book Management System

People are now widely adopting digital assets in various applications, integrating them into almost every aspect of their lives. Electronic books, or e-books, are one of the digital assets that result from the transformation of physical reading material into the digital world. Nowadays, blockchain is used in many industries because it provides immutable and transparent records. E-book publishers may take this opportunity to adopt blockchain technology for e-book data management. However, blockchain storage is limited; thus, storing the e-book files in blockchain is not recommended. A decentralized storage system, such as InterPlanetary Files Systems (IPFS), is an alternative way to store large files like e-books. IPFS can facilitate the storage of e-book files while the metadata is stored in the blockchain. The e-book metadata should be stored in a structured way for effective search and retrieval. E-book metadata could be added, deleted, and updated occasionally. Nevertheless, some data structures often struggle with dynamic collections of records. This paper proposes a linked list-based smart contract on Ethereum that integrates with IPFS for the e-book management system. We demonstrate the implementation of a linked list smart contract for insertion, deletion, update, retrieval, and traversal of the e-book’s metadata. The result shows that a linked list-based smart contract with IPFS could offer a robust solution for e-book data management. This solution provides more opportunities to explore further security and cryptography approaches toward a secure e-book management system.

Intelligent Data Management for Small Business Enhancement

In the industry 4.0 era, there exists a pressing need for intelligent data management solutions to enhance the operations of small businesses. This study introduces a pioneering methodology that harnesses the power of AI-driven analysis of internal voice communications, an often-overlooked source of valuable insights within the small business environment. The research centers on an advanced platform that utilizes the Regularized Bayesian Approach, meticulously tailored for the processing of unstructured and semi-structured data, with a specific focus on internal voice messages. This methodology enables the generation of in-depth insights into employees’ emotional, psychological, and motivational states. Furthermore, the integration of data with a psychometric system enables the production of comprehensive personality evaluations, providing digital portraits for every employee. These portraits offer valuable insights into employee well-being and motivations, particularly beneficial for small businesses with limited HR resources. The potential benefits for small businesses are multifaceted and research-driven, including enhanced employee safety, improved efficiency, advanced risk management, and streamlined HR processes. Additionally, this research underscores the growing relevance and potential of this approach in the Emotion AI market. Through the analysis of voice messages, entities, intent, and relationships between utterances can be discerned, offering a comprehensive view of employee sentiment, loyalty, and satisfaction. This study serves as the foundation for fostering a positive work environment, enhancing productivity, and providing a roadmap for mental health improvement and reduced attrition in small businesses. It contributes to the evolving field of intelligent data management and its applications in enhancing small business operations. Keywords: voice recognition, small business enhancement, emotion AI, artificial intelligence, Bayesian approach

Efficient Zero-Knowledge Proofs for Set Membership in Blockchain-Based Sensor Networks: A Novel OR-Aggregation Approach

Blockchain-based sensor networks offer promising solutions for secure and transparent data management in IoT ecosystems. However, efficient set membership proofs remain a critical challenge, particularly in resource-constrained environments. This paper introduces a novel OR-aggregation approach (where “OR” refers to proving that an element equals at least one member of a set without revealing which one) for zero-knowledge set membership proofs, tailored specifically for blockchain-based sensor networks. We provide a comprehensive theoretical foundation, detailed protocol specification, and rigorous security analysis. Our implementation incorporates optimization techniques for resource-constrained devices and strategies for integration with prominent blockchain platforms. Extensive experimental evaluation demonstrates the superiority of our approach over existing methods, particularly for large-scale deployments. Results show significant improvements in proof size, generation time, and verification efficiency. The proposed OR-aggregation technique offers a scalable and privacy-preserving solution for set membership verification in blockchain-based IoT applications, addressing key limitations of current approaches. Our work contributes to the advancement of efficient and secure data management in large-scale sensor networks, paving the way for wider adoption of blockchain technology in IoT ecosystems.

Data management solutions for real-time analytics in retail cloud environments

In today's rapidly evolving retail landscape, the importance of real-time data analytics has surged, driven by the need for businesses to make informed decisions and enhance customer experiences. This paper explores the critical role of effective data management solutions within cloud environments, focusing on retailers' unique challenges, such as data integration, latency, security, and storage costs. It presents practical solutions, including data partitioning, caching, AI-driven analytics, and the strategic use of data warehousing and data lakes. By adopting hybrid data management models, retailers can optimize their data processes and leverage real-time insights to drive operational efficiency and competitive advantage. This paper offers valuable recommendations for retail businesses and data architects to enhance their cloud-based data management strategies, ensuring they can navigate the complexities of real-time data effectively and sustainably. Keywords: Real-time analytics, Data management, Retail, Cloud environments, Data integration, AI-driven solutions.

A method to enhance privacy preservation in cloud storage through a three-layer scheme for computational intelligence in fog computing

Recent advancements in cloud computing have heightened concerns about data control and privacy due to vulnerabilities in traditional encryption methods, which may not withstand internal attacks from cloud servers. To overcome these issues about the data privacy and control of transfer on cloud, a novel three-tier storage model incorporating fog computing method has been proposed. This framework leverages the advantages of cloud storage while enhancing data privacy. The approach uses the Hash-Solomon code algorithm to partition data into distinct segments, distributing a portion of it across local machines and fog servers, in addition to cloud storage. This distribution not only increases data privacy but also optimises storage efficiency. Computational intelligence plays a crucial role by calculating the optimal data distribution across cloud, fog, and local servers, ensuring balanced and secure data storage.•Experimental analysis of this mathematical mode has demonstrated a significant improvement in storage efficiency, with increases ranging from 30 % to 40 % as the volume of data blocks grows.•This innovative framework based on Hash Solomon code method effectively addresses privacy concerns while maintaining the benefits of cloud computing, offering a robust solution for secure and efficient data management.

A Scalable, Secure, and Efficient Framework for Sharing Electronic Health Records Using Permissioned Blockchain Technology

This paper presents a scalable, secure blockchain-based healthcare system architecture that efficiently manages large patient datasets. DHTs and Skip Lists enable efficient data access, while DPoS and PBFT facilitate parallel transaction processing. Adaptive filters, Radix Trees extended by Merkle Trees, and an immutable blockchain ledger secured by Tendermint consensus ensure data integrity and protection against evolving threats. Threshold Cryptography secures consensus participant selection, and Bulletproofs verify transactions, complying with healthcare regulations. ChaCha20, a symmetric stream cipher, encrypts sensitive data, enhancing performance across devices. ABAC manages access rights, ensuring fine-grained control over data accessibility. This architecture offers a comprehensive, efficient, and secure solution for healthcare data management in blockchain environments.

The dry lab microscopist or prompt microscopist: do we need them?

AbstractIn modern biological microscopy, the explosion of data volume and complexity highlights the urgent need for specialised data management support roles. While traditional microscopy focuses on visual data presentation, the rapid increase in big data acquisition and data mining demands advanced handling and analysis. This gap underscores the need for “dry lab microscopists” or data experts skilled in microscopy data management, software interoperability, and AI-driven solutions. Job markets reflect this demand, pointing to the necessity for dedicated training programs. Integrating these specialists into research institutions is crucial for addressing digital data challenges and maintaining high standards in data integrity and analysis. Their role is essential for advancing research in the data-driven era.

Digital Phenotyping of Mental and Physical Conditions: Remote Monitoring of Patients Through RADAR-Base Platform.

The use of digital biomarkers through remote patient monitoring offers valuable and timely insights into a patient's condition, including aspects such as disease progression and treatment response. This serves as a complementary resource to traditional health care settings leveraging mobile technology to improve scale and lower latency, cost, and burden. Smartphones with embedded and connected sensors have immense potential for improving health care through various apps and mobile health (mHealth) platforms. This capability could enable the development of reliable digital biomarkers from long-term longitudinal data collected remotely from patients. We built an open-source platform, RADAR-base, to support large-scale data collection in remote monitoring studies. RADAR-base is a modern remote data collection platform built around Confluent's Apache Kafka to support scalability, extensibility, security, privacy, and quality of data. It provides support for study design and setup and active (eg, patient-reported outcome measures) and passive (eg, phone sensors, wearable devices, and Internet of Things) remote data collection capabilities with feature generation (eg, behavioral, environmental, and physiological markers). The back end enables secure data transmission and scalable solutions for data storage, management, and data access. The platform has been used to successfully collect longitudinal data for various cohorts in a number of disease areas including multiple sclerosis, depression, epilepsy, attention-deficit/hyperactivity disorder, Alzheimer disease, autism, and lung diseases. Digital biomarkers developed through collected data are providing useful insights into different diseases. RADAR-base offers a contemporary, open-source solution driven by the community for remotely monitoring, collecting data, and digitally characterizing both physical and mental health conditions. Clinicians have the ability to enhance their insight through the use of digital biomarkers, enabling improved prevention, personalization, and early intervention in the context of disease management.

B-031 A new Ready to use Bio-Rad Quality Control to Streamline Daily Workflow

Abstract Background A comprehensive Quality Management System starts with performing a daily Quality Control (Q.C.) process. The daily Q.C. process monitors the Instrument’s optimal performance and detects shifts and trends. However, the existing traditional daily Q.C. process is not only prone to manual errors, but also time consuming, costly and labor intensive. We evaluated a Bio-Rad InteliQTM Load and go Quality Control that comes in a barcoded tube ready to be loaded. We monitored the workflow efficiency gained in comparison to the traditional quality control material that comes in glass vials. Methods We evaluated the new quality control material ready to load on the instrument. This new Q.C. (Bio-Rad InteliQ Assayed Multiqual Control - 3 levels) were processed in parallel with our routine Q.C. (Bio-Rad Assayed Chemistry Lyphocheck vials - 2 levels). We performed the evaluation on Abbott Architect c8000 Chemistry analyzers in our central Laboratory (NGHA-Jeddah). The Q.C. was stored frozen. Before testing it was thawed at room temperature as per the insert instruction. Over a 5 days period, in parallel with our routine quality control, we run the new Q.C. Two times per day for 33 analytes. We monitored the processing time difference between the new and current approaches. We monitored the stability of all the analytes in the tube after Two days and again after Five days. We also monitored the cost effectiveness, labor intensiveness, probability of operator error and the overall improvement in workflow and Turn Around Time. Results Over the evaluation period, we observed the following: - Saving 40 minutes in average per week only for chemistry Q.C (time spent in reconstituting, mixing and aliquoting old Q.C) - 100% reduction in waste of plastic consumables such as pipette and plastic tubes. - Save an average of 1.0 ml dead volume in every 5.0 ml old Q.C. vial (20% of the volume, hence saving 20% of the total Q.C. cost). The dead volume wasted due to aliquoting step using the old Q.C. - 100% Eliminated delays caused by operator error such as wrong reconstitution, inadequate mixing and wrong Q.C. aliquoting. Not needed for the InteliQ. - High score of staff satisfaction. - Stability of the 32 analytes was excellent except that of Co2, which deteriorated by 14% after Two days as indicated in the insert package. Co2 day five result deteriorated by 34%. Conclusions Bio-Rad InteliQ is easy to use. It saves time and cost. When combined with Unity Q.C. data management solution it can streamline Q.C. workflow, eliminate operator errors, reduce T.A.T. and increase the overall laboratory performance. It may also work optimally with Instruments that have on-board fridge for Q.C.

Enhancing Smart Healthcare Networks: Integrating Attribute-Based Encryption for Optimization and Anti-Corruption Mechanisms

This study investigates the feasibility and effectiveness of integrating Attribute-Based Encryption (ABE) into smart healthcare networks, with a particular focus on its role in enhancing anti-corruption mechanisms. The study provides a comprehensive analysis of current vulnerabilities in these networks, identifying potential data security risks. An anti-corruption mechanism is designed to ensure data integrity and reliability. The ABE approach is then empirically compared to other prominent encryption algorithms, such as Identity-Based Encryption, Data Encryption Standard, Advanced Encryption Standard, and Rivest-Shamir-Adleman algorithms. These methods are evaluated based on access latency, data transmission speed, system stability, and anti-corruption capabilities. Experimental results highlight the strengths of the ABE algorithm, demonstrating an average access latency of 31.6 milliseconds, a data transmission speed of 3.56 MB/s, and an average system stability of 98.74%. Furthermore, when integrated into anti-corruption mechanisms, ABE effectively protects against data tampering and misuse, ensuring secure data transmission. Compared to alternative algorithms, ABE offers a more efficient, secure, and stable solution for data management within smart healthcare networks, supported by its robust anti-corruption capabilities. This positions ABE as an optimal choice for safeguarding the integrity and security of healthcare data.

A Comparative Analysis of Microsoft Fabric and Snowflake: Architectural Paradigms and Performance Metrics in Modern Cloud Data Platforms

Abstract: This article presents a comprehensive comparative analysis of two leading cloud data platforms: Microsoft Fabric and Snowflake. As organizations increasingly rely on cloud-based solutions for data management and analytics, understanding the strengths and limitations of these platforms becomes crucial. This article examines the architectural foundations, functional capabilities, performance metrics, and cost considerations of both platforms. We explore Microsoft Fabric's integrated service model within the Azure ecosystem and Snowflake's multi-cluster shared data architecture, assessing their approaches to data integration, storage, analytics, and machine learning integration. Through empirical performance comparisons and an evaluation of scalability mechanisms, we provide insights into the operational efficiencies of each platform. Additionally, we analyze the security features, compliance standards, and pricing models to offer a holistic view of the total cost of ownership. Our findings reveal distinct advantages in Microsoft Fabric's end-to-end integration and Snowflake's performance in multicloud environments, while also highlighting areas for potential improvement in both platforms. This article aims to serve as a valuable resource for organizations navigating the complex landscape of cloud data solutions, offering evidence-based criteria for platform selection aligned with specific business needs and technological ecosystems

HYBRID CLOUD DATABASES FOR BIG DATA ANALYTICS: A REVIEW OF ARCHITECTURE, PERFORMANCE, AND COST EFFICIENCY

The increasing reliance on big data analytics has driven organizations to seek more flexible, scalable, and cost-effective data management solutions, with hybrid cloud databases emerging as a prominent option. Hybrid cloud databases integrate both public and private cloud environments, allowing businesses to balance the scalability and cost advantages of public clouds with the security and control provided by private clouds. This systematic review, conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, examines the architecture, performance, cost efficiency, security, and technological advancements of hybrid cloud databases. The study synthesizes findings from 40 peer-reviewed articles published between 2015 and 2024, focusing on key factors that affect the performance of hybrid cloud databases, such as workload distribution, network latency, and the use of advanced data analytics tools like Hadoop and Spark. The review also explores cost-saving mechanisms, including dynamic resource scaling and pay-as-you-go pricing models, which help organizations reduce infrastructure costs by up to 28%. Additionally, it discusses the security risks and privacy concerns associated with hybrid cloud environments, highlighting the effectiveness of encryption protocols and identity access management systems in mitigating data breaches by as much as 40%. Furthermore, the review identifies emerging trends in the integration of machine learning and artificial intelligence (AI), which have significantly enhanced system automation and resource optimization in hybrid cloud infrastructures, leading to performance improvements of up to 45%.

Application of edge computing and IoT technology in supply chain finance

This study proposes an IoT data management framework based on blockchain and edge computing and conducts detailed simulation experiment design and performance evaluation for the field of supply chain finance. Previous methods often struggled with high latency, limited scalability, and inadequate risk management, and to address these issues, the experimental platform includes traditional centralized systems, distributed systems, blockchain-based systems, and edge computing + blockchain systems. By monitoring indicators such as data processing delay, data transmission delay, data retrieval time, system throughput, and data integrity of each platform, and introducing evaluation formulas for credit risk, operational risk, and market risk, the performance of different systems in processing supply chain financial data is comprehensively analyzed. The experimental results show that the edge computing + blockchain system performs well in data processing efficiency, security, real-time, and system throughput, especially under high load conditions. Our market risk value is reduced to 0.015. This framework improves the efficiency and security of data transmission and effectively reduces credit risk and operational risk, providing an efficient and reliable solution for data management in supply chain finance.

A customizable data management framework for high-repetition-rate high-energy-density science

The high-energy-density (HED) physics community is moving toward a new paradigm of high-repetition-rate (HRR) operation. To fully leverage the scientific power of HRR HED facilities, all of the components of each subsystem (laser, targetry, and performance diagnostics) must be connected and synchronized in a reliable and robust manner while the data acquired are tagged and archived in real time. To this end, GA has begun developing a generalized NoSQL-database framework, the MongoDB repository for information and archiving. An organizational strategy has been developed that shifts HED data organization from a shot-based to a diagnostic-based approach in order to increase archival and retrieval efficiency that lends itself to optimization applications. This work is a first step in pushing HRR HED science toward data management solutions that emphasize machine actionability and aim to stimulate community engagement to define data standards in HED science.

Revealing Halal Certification Oversight Gaps for MSEs through ArcGIS Dashboard Integration

The Indonesian government has developed a comprehensive halal certification program through the Halal Product Assurance Organizing Body (BPJPH). However, the halal certification recording application requires enhancements due to its lack of data elements pertaining to micro and small enterprises (MSEs) with geographical or location references. Geographic Information System (GIS) technology can serve as a solution for data management in the halal industry, encompassing the management, monitoring and analysis of spatial-temporal and real-time data. This research is aimed at developing geospatial technology for industrial halal management within the scope of MSEs. Methods for developing geospatial technology-based management of halal industry applications use analysis, design, development and evaluation through the ArcGIS online platform developed by Environmental Systems Research Institute (Esri). The findings of this research highlight the potential of geospatial technology in the halal industry. It can be effectively utilized to integrate and comprehensively manage halal industry data on a spatial basis, enabling performance monitoring of MSEs, halal product process assistance and the BPJPH. This technology allows for quick analysis of patterns, distributions, trends, associations and regional comparisons, providing valuable insights. Consequently, geospatial technology can support policy recommendations aimed at accelerating the halal certification process by mapping and analyzing relevant data based on geographic locations, facilitating a better understanding of spatial distribution and relationships within the halal industry. This research successfully develops geospatial technology for managing halal industries within MSEs, enabling comprehensive data integration and spatial analysis. The findings highlight its potential to enhance monitoring, support policy recommendations and accelerate the halal certification process.

Ichor: A Python library for computational chemistry data management and machine learning force field development.

We present ichor, an open-source Python library that simplifies data management in computational chemistry and streamlines machine learning force field development. Ichor implements many easily extensible file management tools, in addition to a lazy file reading system, allowing efficient management of hundreds of thousands of computational chemistry files. Data from calculations can be readily stored into databases for easy sharing and post-processing. Raw data can be directly processed by ichor to create machine learning-ready datasets. In addition to powerful data-related capabilities, ichor provides interfaces to popular workload management software employed by High Performance Computing clusters, making for effortless submission of thousands of separate calculations with only a single line of Python code. Furthermore, a simple-to-use command line interface has been implemented through a series of menu systems to further increase accessibility and efficiency of common important ichor tasks. Finally, ichor implements general tools for visualization and analysis of datasets and tools for measuring machine-learning model quality both on test set data and in simulations. With the current functionalities, ichor can serve as an end-to-end data procurement, data management, and analysis solution for machine-learning force-field development.

Advancements in Securing Cloud-Stored Data and Managing Sensitive Information: A Comprehensive Review of Ciphertext-Policy Attribute-Based Encryption (CP-ABE) Techniques

Abstract—The exploring comprehensive review of cutting-edge techniques for securing cloud-stored data and managing sensitive information in the context of smart cities through the application of Ciphertext-Policy Attribute-Based Encryption (CP-ABE). It highlights the innovative integration of blockchain technology with CP-ABE, which introduces a decentralized and tamper- resistant key management system, thereby enhancing the overall security framework in cloud environments where data sharing is prevalent. Introducing an online/offline multi-authority CP- ABE scheme, characterized by hidden policies, offers significant advancements in protecting user attributes and access structures, ensuring that sensitive information remains confidential even during encryption and decryption processes. This dual approach not only fortifies security but also optimizes the efficiency of data- sharing mechanisms. Furthermore, the paper delves into imple- menting hidden sensitive policies and keyword search techniques within smart city infrastructures, which are designed to facilitate secure and efficient data retrieval. These techniques ensure that while data remains accessible to authorized users, privacy is rigorously maintained. Collectively, these approaches represent significant strides in bolstering the security and confidentiality of data in both cloud-based and smart city applications, addressing the growing demand for robust and efficient data management solutions in increasingly interconnected environments. Index Terms—Ciphertext-Policy Attribute-Based Encryption (CP-ABE), Blockchain, Key Management, Decentralized Secu- rity, Cloud-Stored Data, Yolo V7, Explainable AI (XAI), On- line/Offline Multi-Authority Scheme, Hidden policies, Privacy Preservation, Data Protection, Secure Data Management

Environmental Monitoring Data Management and Analysis Based on 5G IoT

As environmental issues become increasingly severe, environmental monitoring has become a crucial measure for protecting and managing ecosystems. The high speed, low latency, and massive connectivity characteristics of 5G technology, combined with the widespread application of IoT technology, provide new solutions for environmental monitoring data management and analysis. This study explores the methods of environmental monitoring data management and analysis based on 5G IoT technology, designs a systematic environmental monitoring framework, and verifies its effectiveness through empirical research. The results show that the environmental monitoring system based on 5G IoT can efficiently, real-time, and reliably collect and analyze environmental data, providing strong support for environmental protection and management.