Improve Data Quality Research Articles

Improving Eye-Tracking Data Quality: A Framework for Reproducible Evaluation of Detection Algorithms.

High-quality eye-tracking data are crucial in behavioral sciences and medicine. Even with a solid understanding of the literature, selecting the most suitable algorithm for a specific research project poses a challenge. Empowering applied researchers to choose the best-fitting detector for their research needs is the primary contribution of this paper. We developed a framework to systematically assess and compare the effectiveness of 13 state-of-the-art algorithms through a unified application interface. Hence, we more than double the number of algorithms that are currently usable within a single software package and allow researchers to identify the best-suited algorithm for a given scientific setup. Our framework validation on retrospective data underscores its suitability for algorithm selection. Through a detailed and reproducible step-by-step workflow, we hope to contribute towards significantly improved data quality in scientific experiments.

Advances in filarial hydrocelectomy reporting in Uganda and Nigeria

IntroductionSurgery for hydrocele is part of the WHO recommended essential package of care for people affected by Lymphatic filariasis (LF). In Uganda and Nigeria, LF programme support for hydrocele surgery began in 2017. In 2021 the two programmes adopted a newly developed smartphone-based application to monitor hydrocele surgery and outcome. The application, referred to as the Hydrocele Tracker, gathers information on surgical registration, preoperative information, intraoperative information, postoperative information and follow-up evaluations. It was developed to improve supervision, understand surgical outcomes and reasons for complications, improve reporting and improve data quality for Ministries of Health with active LF programmes.MethodsHydrocele patient information was collected between October 2021 and May 2023 in Uganda (Lango sub-region, 9 districts) and March 2021 - September 2022 in Nigeria (Benue, Kaduna, Kebbi, Sokoto and Kogi States). Additionally patient feedback surveys were undertaken to elicit patient experience and satisfaction with the hydrocele surgery.ResultsPatient data from 2,911 records in the Hydrocele Tracker are presented.DiscussionThe discussion highlights the implications of the data for LF programming, the value of digitisation and approaches to address data quality.

Using interRAI Assessment for Research: Developing a National Research Agenda in Aotearoa New Zealand

interRAI provides a suite of standardized, validated instruments used to assess health and psychosocial well-being, and to inform person-centered care planning. Data obtained from these standardized tools can also be used at a population level for research and to inform policy, and interRAI is currently used in more than 40 countries globally. We present a brief overview of the use of interRAI internationally within research and policy settings, and then introduce how interRAI is used within the universal public health system in Aotearoa New Zealand (NZ), including considerations relating to Māori, the Indigenous people of NZ. In NZ, improvement in interRAI data utilization for research purposes was called for from aged care, health providers, and researchers, to better use these data for quality improvement and health advancement for New Zealanders. A national research network has been established, providing a medium for researchers to form relationships and collaborate on interRAI research with a goal of translating routinely collected interRAI data to improve clinical care, patient experience, service development, and quality improvement. In 2023, the network members met (hybrid: in-person and online) and identified research priorities. These were collated and developed into a national interRAI research agenda by the NZ interRAI Research Network Working Group. Research priorities included reviewing the interRAI assessment processes, improving methods for data linkage to national data sets, exploring how Indigenous Data Sovereignty can be upheld, as well as a variety of clinically focused research topics. Implications for Practice, Policy, and Research: This appears to be the first time national interRAI research priorities have been formally identified. Priorities identified have the potential to inform quality and clinical improvement activities and are likely of international relevance. The methodology described to cocreate the research priorities will also be of wider significance for those looking to do so in other countries.

The integration algorithm of digital resources in business administration based on cluster analysis

The field of business management involves a large amount of data and information sources, including market data, customer data, supply chain data, etc. In order to quantify and analyze different resources, help enterprises better plan and allocate resources, and improve resource utilization efficiency, a clustering analysis based digital resource integration algorithm for business management is studied. Build a business management digital resource integration framework, including data layer, integration layer, and storage layer, to integrate and store data from different sources of business management databases, thereby facilitating unified management and utilization of digital resources by enterprises. The data layer collects data from different business management databases and stores it in the database according to different sources; The integration layer preprocesses the collected data, simply fixes errors and missing information in the data, and improves data quality. Adopting a feature extraction method based on the projection direction uncorrelation strategy of the labeled power set conversion method, the useful feature information of digital resources in enterprise management can be effectively extracted; Based on the two-step clustering analysis method, business management digital resources are clustered according to similar characteristics to complete the classification and integration of business management digital resources, and improve the efficiency of resource utilization; The storage layer adopts the Security Information Diffusion Algorithm (IDA) storage model to store integrated and classified digital resources managed by enterprises, ensuring data security and effectively preventing data leakage and illegal access. The experimental results show that the digital resource structure of business management integrated by this algorithm is clear, with a data redundancy of less than 8% and a difference of less than 11%. The time consumption for data integration is less than 2.11 minutes, indicating good resource integration ability.

Systems Thinking for Supply Chains: Identifying Bottlenecks Using Process Mapping of a Child Health Intervention in the Democratic Republic of the Congo (DRC)

The quality of supply chains in public health interventions in low- and middle-income countries can determine how effectively a program is able to treat its intended population group and subsequently achieve its health targets. We aimed to disentangle where challenges exist hierarchically and administratively through the application of process mapping to the supply chain of an integrated community case management (iCCM) intervention in the Democratic Republic of the Congo (DRC). We conducted a document review, semi-structured key informant interviews, and focus group discussions with program agents involved in supply chain processes of the child health intervention. Enterprise architecture was used to map the intervention’s supply chain and its participatory actors, and detailed bottlenecks of the chain through the application of a health systems framework. The results of this study will be used to inform a system dynamics model of the supply chain of iCCM in DRC. The greatest bottlenecks leading to stockouts at the community level occurred upstream (from national to province and from zone to health facility). While the use of local procurement processes was partially attempted to strengthen systems, parallel supply chain activities compromised sustainable system integration and development. Initial delays in stock dispensation were due to international procurement at the supplier, inducing a trickle-down effect. Inadequate quantification of supply needs and subsequent insufficient product procuration were the single most important steps that led to stockouts. This study demonstrated that the community health supply chain would be most impacted by improvements made in processes at the highest administrative strata, while exposing its delicate dependence on activities at the lowest levels. Visibility of inventory at all levels and improved data quality and use through a transparent tracking system have the potential to significantly reduce stockouts. Future interventions should take care to not develop parallel processes or exclude local health system agents to avoid disruption and ensure sustainable health outcome gains. Causal loop studies and system dynamics can further identify the systems interactions and relationships and their underlying causal mechanisms in need of intervention.

Improved data quality and statistical power of trial-level event-related potentials with Bayesian random-shift Gaussian processes

Studies of cognitive processes via electroencephalogram (EEG) recordings often analyze group-level event-related potentials (ERPs) averaged over multiple subjects and trials. This averaging procedure can obscure scientifically relevant variability across subjects and trials, but has been necessary due to the difficulties posed by inference of trial-level ERPs. We introduce the Bayesian Random Phase-Amplitude Gaussian Process (RPAGP) model, for inference of trial-level amplitude, latency, and ERP waveforms. We apply RPAGP to data from a study of ERP responses to emotionally arousing images. The model estimates of trial-specific signals are shown to greatly improve statistical power in detecting significant differences in experimental conditions compared to existing methods. Our results suggest that replacing the observed data with the de-noised RPAGP predictions can potentially improve the sensitivity and accuracy of many of the existing ERP analysis pipelines.

Retraction Note: Improvement of seismic data quality and recognition of fault discontinuities through seismic data conditioning applications: a case study of Issaran oil field, Gulf of Suez, Egypt

Retraction Note: Improvement of seismic data quality and recognition of fault discontinuities through seismic data conditioning applications: a case study of Issaran oil field, Gulf of Suez, Egypt

Particulate matter annual emission factors for dairy facilities and cattle feedlots of Texas, USA

The direct emission of inhalable particulate matters (PM2.5 and PM10) from agricultural sources imposes major concern to human health. It is imperative to keep regular update of the pollutant emissions to scrutinize the sources and take necessary preventive measures. Two representative animal feeding operations, one dairy and one feedlot of Texas were selected during the year 2021 for summer and winter sampling using programmable Federal Reference Method (FRM) samplers and Texas A&M University (TAMU) designed samplers. Additionally, for quick and simple emission concentration determination, handheld particle counters were utilized parallel to the FRM and TAMU designed samplers. The air quality index showed that the PM2.5 pollutant fell into ‘Moderate’ category for about 20.93% times and 13.89% times in the dairy and the feedlot respectively. The maximum mean of daily PM10 concentration observed in the Feedlot during the summer was 1676.3 μg/m3 which can be aggravating. The correlation developed using the emission data from the FRM and handheld samplers demonstrated linear relationship and 0.66 of goodness of fit. During the summer, maximum mean PM2.5 emission factors (EF) of 0.84 kg 1000 hd−1 d−1 and 8.09 kg 1000 hd−1 d−1 were determined for the dairy and for the feedlot respectively. However, the winter and the summer PM10 EF for the dairy did not differ significantly (P > 0.05). Although there has not been enough record of PM2.5 annual EF, the current study PM10 annual EF was benchmarked with some of the previously published EFs. This investigation on the air pollutant EFs from the animal feeding operations can greatly benefit in preventing the pollution and updating the most recent EFs. However, continuous monitoring and further research are suggested to improve data quality and develop strategic management practices.

Predictive health monitoring: Leveraging artificial intelligence for early detection of infectious diseases in nursing home residents through discontinuous vital signs analysis

This research addresses the problem of detecting acute respiratory, urinary tract, and other infectious diseases in elderly nursing home residents using machine learning algorithms. The study analyzes data extracted from multiple vital signs and other contextual information for diagnostic purposes. The daily data collection process encounters sampling constraints due to weekends, holidays, shift changes, staff turnover, and equipment breakdowns, resulting in numerous nulls, repeated readings, outliers, and meaningless values. The short time series generated also pose a challenge to analysis, preventing the extraction of seasonal information or consistent trends. Blind data collection results in most of the data coming from periods when residents are healthy, resulting in excessively imbalanced data. This study proposes a data cleaning process and then builds a mechanism that reproduces the basal activity of the residents to improve the classification of the disease. The results show that the proposed basal module-assisted machine learning techniques allow anticipating diagnostics 2, 3 or 4 days before doctors decide to start treatment with antibiotics, achieving a performance measured by the area-under-the-curve metric of 0.857. The contributions of this work are: (1) a new data cleaning process; (2) the analysis of contextual information to improve data quality; (3) the generation of a baseline measure for relative comparison; and (4) the use of either binary (disease/no disease) or multiclass classification, differentiating among types of infections and showing the advantages of multiclass versus binary classification. From a medical point of view, the anticipated detection of infectious diseases in institutionalized individuals is brand new.

An ORL-DLNN and IFIM-MST Framework for Data Quality Improvement in Modern Master Data Management

The process of generating and maintaining a master record for each person in a business is called Master Data Management (MDM). But, during Modern Master Data Management (MMDM), the prevailing methodologies did not focus on data loss. Hence, an Optimal with Recover Layer-based Deep Learning Neural Network (ORL-DLNN) and Indexed Fisher Information Matrix-based Minimum Spanning Tree-based Modern Master Data Management (IFIM-MST-based MMDM) is proposed in this paper. The source data is taken and pre-processed during training. Then, the products are clustered, and the attributes are extracted. Thereafter, by utilizing Dynamic Function-based Gold Rush Optimization (DF-GRO), the optimal attributes are selected. Afterward, for performing format classification with error detection, ORL-DLNN is introduced. Then, by utilizing the Reconciliation Rule (RR), the detected errors are corrected. After that, by utilizing the IFIM-MST, the text records are stored in the primary drive and the video and images are stored in the external drive. The details are entered by the information stewards into the classifier during testing. The data will be corrected if correction is present. Then, for verifying whether the data must be stored or not, some conditions are checked. Hence, the outcomes illustrated that the proposed system obtained a high accuracy of 98.3%, thus outperforming the existing techniques

TabFedSL: A Self-Supervised Approach to Labeling Tabular Data in Federated Learning Environments

Currently, self-supervised learning has shown effectiveness in solving data labeling issues. Its success mainly depends on having access to large, high-quality datasets with diverse features. It also relies on utilizing the spatial, temporal, and semantic structures present in the data. However, domains such as finance, healthcare, and insurance primarily utilize tabular data formats. This presents challenges for traditional data augmentation methods aimed at improving data quality. Furthermore, the privacy-sensitive nature of these domains complicates the acquisition of the extensive, high-quality datasets necessary for training effective self-supervised models. To tackle these challenges, our proposal introduces a novel framework that combines self-supervised learning with Federated Learning (FL). This approach aims to solve the problem of data-distributed training while ensuring training quality. Our framework improves upon the conventional self-supervised learning data augmentation paradigm by incorporating data labeling through the segmentation of data into subsets. Our framework adds noise by splitting subsets of data and can achieve the same level of centralized learning in a distributed environment. Moreover, we conduct experiments on various public tabular datasets to evaluate our approach. The experimental results showcase the effectiveness and generalizability of our proposed method in scenarios involving unlabeled data and distributed settings.

Concept of a Satellite Cross-Calibration Radiometer for In-Orbit Calibration of Commercial Optical Satellites

The satellite Earth observation (EO) sector is burgeoning with hundreds of commercial satellites being launched each year, delivering a rich source of data that could be exploited for societal benefit. Data streams from the growing number of commercial satellites are of variable quality, limiting the potential for their combined use in science applications that need long time-series data from multiple sources. The quality of calibration performed on optical sensors onboard many satellite systems is highly variable due to calibration methods, sensor design, mission objective, budget, or other operational constraints. A small number of currently operating well-characterised satellite systems with onboard calibration, such as Landsat-8/9 and Sentinel-2, and planned future missions, like the NASA Climate Absolute Radiance and Refractivity Observatory (CLARREO) Pathfinder, the European Space Agency (ESA)’s Traceable Radiometry Underpinning Terrestrial and Helio Studies (TRUTHS), and LIBRA from China, are considered benchmarks for optical data quality due to their traceability to international measurement standards. This paper describes the concept of a space-based transfer calibration radiometer called the Satellite Cross-Calibration Radiometer (SCR) that would enable the calibration parameters from satellites such as Landsat-8/9, Sentinel-2, or other benchmark systems to be transferred to a range of commercial optical EO satellite systems while in orbit. A description of the key characteristics of the SCR to successfully operate in orbit and transfer calibration from reference systems to client systems is presented. A system like the SCR in orbit could complement SI-Traceable satellites (SITSats) to improve data quality and consistency and facilitate the interoperable use of data from multiple optical sensor systems for delivering higher returns on the global investment in EO.

Standardizing Substrate Selection: A Strategy toward Unbiased Evaluation of Reaction Generality.

With over 10,000 new reaction protocols arising every year, only a handful of these procedures transition from academia to application. A major reason for this gap stems from the lack of comprehensive knowledge about a reaction's scope, i.e., to which substrates the protocol can or cannot be applied. Even though chemists invest substantial effort to assess the scope of new protocols, the resulting scope tables involve significant biases, reducing their expressiveness. Herein we report a standardized substrate selection strategy designed to mitigate these biases and evaluate the applicability, as well as the limits, of any chemical reaction. Unsupervised learning is utilized to map the chemical space of industrially relevant molecules. Subsequently, potential substrate candidates are projected onto this universal map, enabling the selection of a structurally diverse set of substrates with optimal relevance and coverage. By testing our methodology on different chemical reactions, we were able to demonstrate its effectiveness in finding general reactivity trends by using a few highly representative examples. The developed methodology empowers chemists to showcase the unbiased applicability of novel methodologies, facilitating their practical applications. We hope that this work will trigger interdisciplinary discussions about biases in synthetic chemistry, leading to improved data quality.

Current benefits and future possibilities with digital field reporting

Mobile phones and tablets enable contractors to digitally collect large amounts of production remarks and facilitate the acquisition. The increased data access and machine learning techniques allow the construction industry to take a significant step forward in shifting from implicit to explicit knowledge. However, this step requires both standardisation and data quality assurance combined with project incitements ensuring continuous data collection. Therefore, this study examines the current data quality and standardisation of inspection data generated using the production software Dalux Field, mining a dataset of more than 95000 production issues. Additionally, a survey of production software users assesses project and project member benefits and future possibilities with digital inspection reporting. The results show considerable benefits with digital inspection reporting, such as time savings, cost reductions and increased general quality control. However, the standardisation in reporting between projects and team members is low. Finally, this paper suggests methods for improving data quality and standardization for automation of the data analysis, allowing new projects in project-based organisations to benefit from previous project experiences.

A data mining-based framework to define optimal operation strategies for in-duct ultraviolet germicidal irradiation systems for different climatic conditions

This paper presents a novel data mining framework for optimizing the operation of in-duct Ultraviolet germicidal irradiation (UVGI) systems employed for air disinfection within air handling units. The performance of the UVGI system is influenced by the operating conditions of air handling units. As a result, under favourable operating conditions, the system produces excessive Ultraviolet C energy beyond the required disinfection levels, causing energy waste and accelerating the degradation of heating, ventilation, and air conditioning (HVAC) components. The framework aims to improve energy efficiency and mitigate material degradation within HVAC components by systematically adjusting UVGI system output based on changing operating conditions of air handling units throughout the year using advanced data mining techniques. The methodology involves several steps: Firstly, data pre-processing is performed to improve data quality. Then, the K-means clustering algorithm is applied to discover patterns in operational data. Subsequently, an effective performance index and baseline are defined to identify energy-saving opportunities within each cluster. Finally, the optimal operation strategy is defined based on the determined baseline. The framework is extended to five distinct climate zones to derive region-specific operation strategies. Results from a case study in a small office building in Montreal and an extension to five distinct climate zones in Canada demonstrate energy savings from 8% to 37%. This study offers a promising approach to optimize UVGI system operation, with potential applications in various energy systems in buildings requiring adaptive operational conditions for optimal performance.

Does it Pay to Send Multiple Pre-Paid Incentives? Evidence from a Randomized Experiment

To encourage survey participation and improve sample representativeness, the Survey of Consumer Finances (SCF) offers an unconditional pre-paid monetary incentive and separate post-paid incentive upon survey completion. We conducted a pre-registered between-subject randomized control experiment within the 2022 SCF, with at least 1,200 households per experimental group, to examine whether changing the pre-paid incentive structure affects survey outcomes. We assess the effects of: (1) altering the total dollar value of the pre-paid incentive (“incentive effect”), (2) giving two identical pre-paid incentives holding the total dollar value fixed (“reminder effect”), and (3) offering multiple pre-paid incentives of different amounts holding the total dollar value fixed (“slope effect”) on survey response rates, interviewer burden, and data quality. Our evidence indicates that a single $15 pre-paid incentive increases response rates and maintains similar levels of interviewer burden and data quality, relative to a single $5 pre-paid incentive. Splitting the $15 into two pre-paid incentives of different amounts increases interviewer burden though lengthening time in the field without improving response rates, reducing the number of contact attempts needed for a response, or improving data quality, regardless of whether the first pre-paid is larger or smaller than the second.

Dimensions of data sparseness and their effect on supply chain visibility

Supply chain visibility concerns the ability to track parts, components, or products in transit from supplier to customer. The data that organizations can obtain to establish or improve supply chain visibility is often sparse. This paper presents a classification of the dimensions of data sparseness and quantitatively explores the impact of these dimensions on supply chain visibility. Based on a review of supply chain visibility and data quality literature, this study proposes to characterize data sparseness as a lack of data quality across the entire supply chain, where data sparseness can be classified into three dimensions: noise, bias, and missing values. The quantitative analysis relies on a stylized simulation model of a moderately complex illicit supply chain. Scenarios are used to evaluate the combined effect of the individual dimensions from actors with different perspectives in the supply chain, either supply or demand-oriented. Results show that when a data sparseness of 90% is applied, supply chain visibility reduces to 52% for noise, to 65% for bias, and to 32% for missing values. The scenarios also show that companies with a supply-oriented view typically have a higher supply chain visibility than those with a demand-oriented view. The classification and assessment offer valuable insights for improving data quality and for enhancing supply chain visibility.

What do veterinary professionals and clients want? Using discrete choice methods in veterinary research to determine preferences.

Discrete choice methods (DCMs) are a suite of research techniques for identifying individual preferences using choice information. Widely utilized by other fields yet rarely employed in veterinary research, DCMs have tremendous potential to improve veterinary healthcare by understanding and incorporating owner and veterinary professionals' (encompassing veterinarians, veterinary clinicians, technicians, receptionists, attendants, etc) preferences to optimize the care continuum. DCMs have several advantages over other stated preference methods, such as ranking and ratings, including improved data quality and actionability. However, they are not a panacea, and limitations that may affect DCMs' application to the veterinary field are outlined alongside realistic mitigation strategies. The information provided aims to increase awareness of DCMs and their utility in veterinary research and encourage greater uptake as a more robust method for measuring preferences.

Enhancing Industrial IoT Security: Utilizing Blockchain-Assisted Deep Federated Learning for Collaborative Intrusion Detection

The Industrial Internet of Things (IIoT) is a rapidly evolving features with multiple applications, including critical infrastructure. Privacy policies are required to preserve the protection of user data in the threat intelligence community. Blockchain is a modern technology which used recently to provide more secure storage and efficiency. In this research, Blockchain Assisted Deep Federated Learning (BC_DFL) system is used to detect intruders. The three key processes used in the proposed intrusion detection architecture are data collection, pre-processing and intrusion detection. Data normalization, reduction, cleaning and transformation are used in pre-processing to remove extraneous information and improve data quality. This pre-processed data is sent to the Blockchain Assisted Deep Federated Learning (BC_DFL) system for intrusion detection. To detect intruders, the federated learning-based Capsule Auto-Encoder (FL_CAE) architecture first learns the properties from the inputs. Blockchain technology (BCTech) is not only used for storage but also improves security by eliminating the possibility of threatening node and individual server failure. The ToN_IoT and UNSW-NB15 data sets are used in the implementation and performance evaluation study. The proposed model is evaluated using existing in the Results section. In the UNSW-NB15 dataset, proposed model achieved an accuracy, precision, recall and F1 score of 97.26%, 97.28%, 96.89% and 96.96% respectively as well as in ToN_IoT data set proposed model achieved an accuracy, precision, recall and F1 score of 95.74%, 99.54 %, 99.49% and 99.24%, respectively. The execution of the proposed approach takes 2.31 seconds in the UNSW-NB15 dataset and 1.66 seconds in the ToN_IoT dataset. Blockchain offers a transparent and impenetrable ledger for transaction recording and verification. Within the framework of cooperative intrusion detection, it guarantees a secure and reliable exchange of threat intelligence and detection models between various users of IIoT ecosystem.

Analysis and Modelling of Structured Data with Automatic Data Analysis Web Application

Abstract: This paper introduces a novel web application designed for automated analysis and modelling of structured data. The application eliminates the need for user coding, allowing users to interact with the data through a user-friendly interface. By uploading a structured dataset, users can leverage the application's functionalities to improve data quality. These functionalities include, but are not limited to, data cleaning and pre-processing. One of the key strengths of this application is its ability to automatically generate machine learning models based on the uploaded data. Furthermore, the application takes the automation a step further by training the generated models without requiring user intervention. This eliminates the need for data science expertise, making the power of machine learning accessible to a wider audience. This paper details the application's architecture and functionalities, along with an evaluation of its effectiveness in data analysis and model building. The paper discusses the application's potential impact on democratizing data science and its limitations for future research directions.