Efficient Data Management System Research Articles

Content-Defined Chunking Algorithms in Data Deduplication: Performance, Trade-Offs and Future-Oriented Techniques

In the digital era, the exponential growth of data presents significant challenges for storage efficiency and processing speed. This paper reviews Content-Defined Chunking (CDC), a cornerstone in data deduplication technology, aimed at addressing these challenges. We systematically examine various CDC algorithms, categorising them into hashing-based and hash-less methodologies, and evaluating their performance in deduplication processes. Through a critical analysis of existing literature, the study identifies the balance between chunking speed and deduplication efficacy as a pivotal area for enhancement. Our findings reveal the need for innovative CDC algorithms to adapt to the evolving data landscape, proposing future research directions for improving storage and processing solutions. This work contributes to the broader understanding of data deduplication techniques, offering a pathway towards more efficient data management systems.

Optimalisasi Dashboard Pemesanan Makanan Online Menggunakan Looker dan JavaScript

Advances in information technology have enabled rapid development in internet-based services, including online food ordering applications. This growth demands efficient data management and analysis systems to improve user experience and operational performance. This research focuses on developing an optimal online food ordering dashboard using Looker and JavaScript. Research methods include needs identification, literature study, needs analysis, design, and implementation. The research results show that the dashboard developed is able to manage and analyze order data effectively, identify trends, predict customer needs, and increase operational efficiency. This dashboard visualizes important information such as number of orders based on age, gender, income, as well as customer behavior analysis. In doing so, service providers can gain better insight into consumer behavior and ordering trends, supporting more informed and strategic decision making. The results of this study contribute to the literature on the use of data visualization technologies in the online food service sector.

Digitizing operating theater data in resource-limited settings: Understanding surgical care delivery post-implementation at Tanzanian referral hospital.

Digitizing surgical data infrastructure is critical for policymakers to make informed decisions. The implementation of the first web-based operating theater (OT) recordings at Muhimbili Orthopedic Institute (MOI) represents significant advancements in data management for Tanzania. This study aims to share post-platform implementation outcomes, challenges, and insights gained offering guidance to settings facing similar data repository challenges. In July 2023, after training clinicians, the platform was deployed at MOI operating theaters (OTs) to facilitate prospective data entry following procedures, ensuring timely updates of perioperative outcomes. Semi-structured interviews were conducted with key stakeholders to gather insights into the platform's functionality and efficient data management systems. We presented data from August 2023 to February 2024 along with platform insights. Over 4449 procedures were conducted, comprising 1321 emergencies and 3128 electives, with orthopedics/trauma accounting for the majority (3606). Trauma-related emergencies (921) predominate among interventions. General anesthesia was prevalent; 60.56% in emergencies and 44.51% in electives. Orthopedics/trauma utilized 90.91% of assigned operating days in electives, while neurosurgery utilized 93.39% (p<0.011). The cancellation rate was 7.5%, primarily due to emergency interferences (32%). Of procedures, 96.76% were discharged, while 2.81% died. Challenges encountered during platform implementation included securing local support, integrating technology, and navigating administrative adjustments. Lessons learned emphasized continuous communication for stakeholder buy-in and training for platform familiarity. The web-based OT recordings at MOI succeeded with local support and showed promise for wider scalability. To ensure sustainability, ongoing follow-up, monitoring of platform functionality, local funding establishment, and strengthening global partnerships are recommended.

CUPID: An efficient spatio-temporal data engine

In the IoT era, abundant spatio-temporal data is generated from various devices thanks to the prevalence of positioning techniques. Due to a lack of effective systems to manipulate the data, advanced scalable and efficient data management systems are necessary to support more and more urban applications.We propose Cupid, which is powered by scholars from Chongqing University and Guangzhou Urban Planning and Design Survey Research Institute, an efficient spatio-temporal Data engine. It extends JUST by introducing many functionalities to make it more applicable to deployment, and can efficiently manage large-scale spatio-temporal data. In Cupid, Apache HBase is utilized as the storage, GeoMesa serves as the spatio-temporal data indexing tool, and Apache Spark acts as the execution engine. We introduce many optimizations to ensure usability and reduce computational overhead. To make Cupid easy to use, we design and implement a SQL-like query language. Furthermore, to deploy the system as a PaaS while speeding up the execution, we leverage Apache Livy, a service that enables easy interaction with a Spark cluster over REST interfaces, together with gRPC to effectively collect results. Extensive experiments illustrate that Cupid outperforms other state-of-the-art spatio-temporal big data management systems in terms of efficiency and scalability.

Damage identification in sandwich structures using Convolutional Neural Networks

The increasing complexity of structures and materials, coupled with ever more stringent demands for safety and cost reduction in maintenance operations, has driven the need to develop advanced techniques for structural integrity monitoring, known as Structural Health Monitoring (SHM). In this context, this study investigates the use of image processing techniques of mode shapes in a composite sandwich panel, employing various Convolutional Neural Network (CNN) models, with the purpose of identifying damage. The main objective is to classify the type of damage, whether it is in the core, at the interface, or in the laminate, followed by the precise location of the flaw and determination of damage dimensions in terms of length and width. To achieve these goals, the finite element method was used to create a representative database, and subsequently, an efficient data management system and model implementation were established to optimize computational costs. The results obtained show that, in the classification task, a high accuracy in damage type identification (99%) was achieved, even when images had high levels of noise. Regarding the regression task for damage localization, satisfactory results were obtained, while the dimensioning of damage length proved acceptable, albeit with some limitations, and inclination identification presented challenges. This study represents a significant advancement in the field of SHM and explores both the advantages and limitations of using Convolutional Neural Networks for this purpose. The results obtained provide valuable insights for the practical application of these techniques in the detection and assessment of damage in sandwich structures, thus contributing to enhancing the safety and efficiency of maintaining such structures.

A blockchain‐based secure framework for data management

AbstractData management is a crucial requirement due to the autonomous and constrained nature of Unmanned Aerial Vehicles (UAVs), Internet of Things (IoTs), and the aviation domain. The autonomous and restricted nature of these sectors increases the need for a shared, distributed database, strong access control management, consensus in autonomous decision‐making, and effective communication across diverse protocols and devices. This research presents a comprehensive approach and offers a new viewpoint to the field of blockchain while establishing a fundamental baseline for future improvements in data management systems and addressing the shortcomings of previously proposed existing frameworks in order to fulfill the complex needs of secure data management. This study contributes to the advancement of secure and efficient data management systems by implementing robust data monitoring for error detection, ensuring data integrity, and enabling encrypted or anonymous data sharing based on sensitivity levels. Additionally, the integration of diverse devices, enforcement of immutable regulations compliance, and development of permissioned blockchain systems for identity management further enhance the system's capabilities, offering comprehensive solutions for modern data management challenges. In the tests, the proposed framework showed increased successful transactions in all rate controllers. Besides, effect of the validator number on throughput and latency is tested and analyzed thoroughly.

Best practices for data management and sharing in experimental biomedical research.

Effective data management is crucial for scientific integrity and reproducibility, a cornerstone of scientific progress. Well-organized and well-documented data enable validation and building on results. Data management encompasses activities including organization, documentation, storage, sharing, and preservation. Robust data management establishes credibility, fostering trust within the scientific community and benefiting researchers' careers. In experimental biomedicine, comprehensive data management is vital due to the typically intricate protocols, extensive metadata, and large datasets. Low-throughput experiments, in particular, require careful management to address variations and errors in protocols and raw data quality. Transparent and accountable research practices rely on accurate documentation of procedures, data collection, and analysis methods. Proper data management ensures long-term preservation and accessibility of valuable datasets. Well-managed data can be revisited, contributing to cumulative knowledge and potential new discoveries. Publicly funded research has an added responsibility for transparency, resource allocation, and avoiding redundancy. Meeting funding agency expectations increasingly requires rigorous methodologies, adherence to standards, comprehensive documentation, and widespread sharing of data, code, and other auxiliary resources. This review provides critical insights into raw and processed data, metadata, high-throughput versus low-throughput datasets, a common language for documentation, experimental and reporting guidelines, efficient data management systems, sharing practices, and relevant repositories. We systematically present available resources and optimal practices for wide use by experimental biomedical researchers.

Big Data Analytics for the Moroccan Water Actors: Towards an Adoption Model

This paper delves into the adoption of Big Data Analytics (BDA) within the Moroccan water management sector, specifically examining data collection, management flow, and decision-making processes. Through insightful interviews with representatives from the public sector responsible for water management, this study investigates the pivotal role of effective data governance in optimizing water resources[1]. Given the centrality of natural resources, particularly water, in national development strategies, the imperative for robust water governance is underscored. The impact of climate change further highlights the need for sufficient data collection and an efficient data management system[2]. In the management of natural resources, the accumulation of extensive data necessitates the application of BDA technology for streamlined processing. While BDA provides a solution for optimal data utilization, this article emphasizes the importance of maintaining control over processing and decision-making to ensure the success of functional and technical designs. The outcomes contribute to the formulation of an adoption model for BDA in the Moroccan water sector, facilitating well-informed and strategic decision-making for sustainable water resource management.

Pembangunan Website Informasi Kepegawaian Pada UPTD Teknologi Komunikasi Dan Informasi Pendidikan

Information and Communication Technology (ICT) is a crucial element in modern life, encompassing the management, manipulation, and transfer of information across media platforms. UPTD Teknologi Komunikasi dan Informasi Pendidikan is a Structurally Integrated Service Unit under the East Kalimantan Provincial Education Office. UPTD plays a strategic role in providing technology-related services in the field of education. However, UPTD faces challenges in the inefficient and slow management of employee data, with 28 employees whose personnel data remains unoptimized. To enhance efficiency, the author plans to develop a web-based personnel information system. The problem statement includes inadequate system support, resulting in slow employee search and report generation processes. The objective of this system is to design an efficient, precise, and accurate personnel data management system, aiming to improve work productivity within UPTD Teknologi Komunikasi dan Informasi Pendidikan. The outcome of this initiative is the design of a website serving as an information portal for UPTD profile, developed using the Content Management System (CMS) WordPress.

A GEO-DATABASE FOR 3D-AIDED MULTI-EPOCH DOCUMENTATION OF BRIDGE INSPECTIONS

Abstract. The recent collapse of bridges in Italy has prompted numerous studies on monitoring and maintenance. Many structures in Italy have been in service for over 50 years, necessitating new approaches to ensure their safety. To address this issue, Italy's Consiglio Superiore dei Lavori Pubblici (Superior Council of Public Works) has developed the Guidelines for Risk Classification and Management, proposing a multi-level approach to bridge management within a complex geomorphological environment. The guidelines outline a multi-level process that includes surveying the structures, conducting detailed inspections, and assigning risk classes based on hazard, exposure, and vulnerability. Current inspection processes are time-consuming and costly. Therefore, alternative monitoring technologies are crucial. Unmanned aerial vehicles equipped with cameras, laser technologies, and GPS systems offer flexible and cost-effective solutions for visual inspection. These technologies enable the collection of both quantitative and qualitative data, such as size, material properties, and overall condition. In this context, efficient data management and exploration systems are necessary to handle the vast amount of geo-referenced information. Multi-epoch databases play a crucial role in documenting the conditions of bridges and supporting a maintenance and structural health monitoring workflow. These databases can be utilized within a Bridge Management System to aid road managers in decision-making processes. Additionally, 3D exploration platforms provide visual analysis and highlight areas of interest within the structure. This work presents a multi-epoch geo-database that adheres to the Italian guidelines, offering optimized data management and queryability for 2D and 3D information. The entire process is designed using open-source and reproducible solutions.

Efficient and secure medical big data management system using optimal map-reduce framework and deep learning

Efficient and secure medical big data management system using optimal map-reduce framework and deep learning

Secure And Efficient Access Control Over Blockchain PHR Cloud Storage System

A novel architecture of blockchain-based secured electronic health record (EHR) management system is presented in this paper. Electronic health data record-keeping in cloud-based storage systems always pose a threat to information security. Intruders can delete EHR of patients, giving benefits to insurance companies or hiding medical malpractices. A secured EHR management system is required that would essentially solve such issues. The blockchain is an emerging technology that can be adapted to develop a secured and an efficient data management system. Storing and sharing health records through electronic systems pose security risks. However, establishing a new blockchain based system replacing the existing system is expensive. In our proposed architecture, we introduced an integration mechanism, named as the blockchain handshaker, between the existing cloud based EHR management system and public blockchain network to develop a secured health record management system. A blockchain-based protocol that secures health records, addressing all of the main security and complementary properties defined in current regulations. A prototype to provide evidence on the feasibility of the proposed concept is presented in this paper. The performance assessments demonstrate the efficiency of our proposed solution in terms of computation, communication, and storage.

Compact Back-End Electronics with Temperature Compensation and Efficient Data Management for In Situ SiPM-Based Radiation Detection.

A compact back-end interface for silicon photomultipliers (SiPMs) implementing Zener diode-based temperature compensation has been developed for the remote detection of beta and gamma radiation. Remote detection is facilitated by the development of an efficient data management system utilising MySQL database storage for recording periodic spectra data for wireless access over a private Wi-Fi network. A trapezoidal peak shaping algorithm has been implemented on an FPGA for the continuous conversation of pulses from the SiPM, signifying the detection of a radiological particle, into spectra. This system has been designed to fit within a 46 mm cylindrical diameter for in situ characterization, and can be attached to one or more SiPMs used in conjunction with a range of scintillators. LED blink tests have been used to optimise the trapezoidal shaper coefficients to maximise the resolution of the recorded spectra. Experiments with an array of SiPMs integrated with a NaI(Tl) scintillator exposed to sealed sources of Co-60, Cs-137, Na-22 and Am-241 have shown that the detector achieves a peak efficiency of 27.09 ± 0.13% for a gamma peak at 59.54 keV produced by Am-241, and a minimum energy resolution (Delta E/E) of 4.27 ± 1.16% for the 1332.5 keV gamma peak from Co-60.

Online Presence System Design Based On Data Granted Validity

The presence of employees in an agency is very important for recording attendance accurately and avoiding fraud in the manual attendance process. Therefore, designing an online presence system is the right solution to overcome this problem. A good online presence system must have high accuracy and be integrated with an efficient data management system. One way to improve the accuracy of the online attendance system is to use the data validation method provided. The manual attendance system used by the company suffers from inaccuracies and fraud. Therefore, this study aims to overcome this problem by designing and implementing an online presence system based on the validity of the data provided. Information system development methods are used to design online presence systems that use blockchain technology to ensure data validity and reduce the risk of fraud. In conclusion, an online attendance system based on the validity of the data provided can be an effective and efficient alternative solution for institutions that still use manual attendance systems.

RNA‐seq‐driven expression analysis to investigate cardiovascular disease genes with associated phenotypes among atrial fibrillation patients

RNA‐seq‐driven expression analysis to investigate cardiovascular disease genes with associated phenotypes among atrial fibrillation patients

The Immunization Data Quality Assessment, Sana’a Capital, 2021

Background The Expanded Program of Immunization (EPI) aims to increase immunization coverage. However, this cannot be achieved without an efficient data management system and without ensuring data quality. Objective We aimed to assess the quality of immunization data at Sana’a capital. Methods The World Health Organization data quality self-assessment tools were used. Three random urban districts and the only rural district (Bani-Al Hairth) at Sana’a capital were selected. From each district, one-third of the public health facilities (HFs) that were providing EPI services were randomly selected. Accuracy ratios (ARs), discrepancy levels (DLs), completeness, and timeliness were calculated from tally sheets and reports for Bacillus Calmette-Guerin (BCG) vaccines, third doses of pentavalent-3 (Penta-3) vaccines, and first doses of measles and rubella (MR-1) vaccines. The quality index was assessed for the five components (ie, recording and reporting, archiving, demographic information, core output/analysis, and using data for action) through a prestructured questionnaire. Results While the overall ARs and DLs for BCG, Penta-3, and MR-1 indicated overreporting at the HF level, there was overreporting for BCG and Penta-3 and underreporting for MR-1 at the district level. With regard to the overall quality index, recording and reporting achieved the highest score (90% and 96%, respectively), while using data for action had the lowest score (61% and 78%, respectively) at the HF and district levels. While completeness and timeliness were scored 100% at all HFs, both were inadequate at the Al-Sabain (93% and 99%, respectively) and Bani-Al Hairth (75% and 83%, respectively) districts. Conclusions The findings showed that the quality of immunization data in Sana’a capital’s HFs and districts was inadequate, with weaknesses in using data for action. Furthermore, completeness and timeliness were found to be unsatisfactory at the rural district and one of the urban districts. Ensuring data quality through strengthening the EPI data management system should be prioritized. Larger-scale and regular assessments of the EPI data management system are recommended.

CellDepot: A Unified Repository for scRNA-seq Data and Visual Exploration

CellDepot containing over 270 datasets from 8 species and many tissues serves as an integrated web application to empower scientists in exploring single-cell RNA-seq (scRNA-seq) datasets and comparing the datasets among various studies through a user-friendly interface with advanced visualization and analytical capabilities. To begin with, it provides an efficient data management system that users can upload single cell datasets and query the database by multiple attributes such as species and cell types. In addition, the graphical multi-logic, multi-condition query builder and convenient filtering tool backed by MySQL database system, allows users to quickly find the datasets of interest and compare the expression of gene(s) across these. Moreover, by embedding the cellxgene VIP tool, CellDepot enables fast exploration of individual dataset in the manner of interactivity and scalability to gain more refined insights such as cell composition, gene expression profiles, and differentially expressed genes among cell types by leveraging more than 20 frequently applied plotting functions and high-level analysis methods in single cell research. In summary, the web portal available at http://celldepot.bxgenomics.com, prompts large scale single cell data sharing, facilitates meta-analysis and visualization, and encourages scientists to contribute to the single-cell community in a tractable and collaborative way. Finally, CellDepot is released as open-source software under MIT license to motivate crowd contribution, broad adoption, and local deployment for private datasets.

DRONE-BASED CONTAINER CRANE INSPECTION: CONCEPT, CHALLENGES AND PRELIMINARY RESULTS

Abstract. Container crane inspection is a very important task to maintain their uninterrupted operation. Nevertheless, this is a costly and time-consuming activity if performed manually. Recently, image-based detection of surface damages or changes using drones has gained increasing interest in industry; especially when objects of interest have a complex structure like container cranes. One main aim of this paper is a single-epoch image analysis which will also serve later for multi-epoch processing. It provides reliable information about current defects that may lead to big damages if not inspected by experts. Naïve Bayes classifier is employed to classify the images in different classes of which critical defects and especially rust is important. The preliminary results show that the precision on the target class reached about 99%. However, 87% percent recall in this class is not enough and it should be improved for this application.Having a large dataset requires an efficient data management system to provide users and decision makers with the information needed. In addition, in order to foster full automation, the aforementioned image analysis component should have a direct connection to the database and thus is able to query image and semantic information. We therefore introduce the second aim of our research, that is a concept for database design. Here, not only the raw data and the final results are integrated but also the intermediate results. At the same time, the database concept is connected to an integrated client interface that allows retrieving data of interest in a virtual globe.

Tunable Q-wavelet based ECG data compression with validation using cardiac arrhythmia patterns

The acquisition of longer electrocardiogram (ECG) signals is an essential step of the recent medical screening and diagnostic procedures. Considering the need of an efficient ECG data management system, this paper presents a novel ECG data compression technique based on the tunable Q-wavelet transform which provides adjustable parameters to achieve good compression performance. The tunable Q-wavelet transform compacts the maximum energy of the signal to fewer transform coefficients. Dead-zone quantization is used to discard the small valued transform coefficients. Further, transform coefficients are rounded-off to nearest integer values and encoded using run-length coding. ECG records of the MIT-BIH arrhythmia database are used to evaluate the performance of the proposed compression technique. The average compression performance obtained in terms of CR, PRD, PRD1, QS, QS1 and SNR are 20.61, 4.43, 6.37, 5.88, 3.46, and 55.93 dB respectively. The proposed technique offers better compression performance than many existing techniques. The proposed technique is also validated using cardiac arrhythmia patterns classification. In the validation phase, tunable Q-wavelet transform based features of 14,878 original cardiac patterns are used to train the support vector machine classifier while testing is performed using features of 26,219 original and reconstructed cardiac patterns separately. It examines the effect of the proposed compression technique on cardiac arrhythmia classification. In the validation part, the overall classification accuracy, sensitivity, and specificity obtained for the proposed compression technique are 98.35%, 95.77%, and 99.19% respectively. It denotes that the proposed technique compresses ECG signal efficiently with preserving diagnostic information very well.

Clearing the Digital Minefield for Optimized Oilfield Production

The success of any digital oilfield project is predicated on the quality of the data structure, acquisition, communication, validation, storage, retrieval, and provenance of the data. Therefore, the mission of the digital oilfield architect is clear: Deliver the right data to the right users at the right time. This task is becoming more difficult because of the sheer variety of sensors, the multitude of data formats, challenges with data communication and synchronization, and the potentially large volume of data produced. While the advent of data analytics has increased the importance of designing fast and efficient data management systems, it is a multifaceted minefield to gather and process data to ensure production is optimized and risk is minimized in both conventional and unconventional operating environments.