Improve Data Quality Research Articles

Systems Thinking for Supply Chains: Identifying Bottlenecks Using Process Mapping of a Child Health Intervention in the Democratic Republic of the Congo (DRC)

The quality of supply chains in public health interventions in low- and middle-income countries can determine how effectively a program is able to treat its intended population group and subsequently achieve its health targets. We aimed to disentangle where challenges exist hierarchically and administratively through the application of process mapping to the supply chain of an integrated community case management (iCCM) intervention in the Democratic Republic of the Congo (DRC). We conducted a document review, semi-structured key informant interviews, and focus group discussions with program agents involved in supply chain processes of the child health intervention. Enterprise architecture was used to map the intervention’s supply chain and its participatory actors, and detailed bottlenecks of the chain through the application of a health systems framework. The results of this study will be used to inform a system dynamics model of the supply chain of iCCM in DRC. The greatest bottlenecks leading to stockouts at the community level occurred upstream (from national to province and from zone to health facility). While the use of local procurement processes was partially attempted to strengthen systems, parallel supply chain activities compromised sustainable system integration and development. Initial delays in stock dispensation were due to international procurement at the supplier, inducing a trickle-down effect. Inadequate quantification of supply needs and subsequent insufficient product procuration were the single most important steps that led to stockouts. This study demonstrated that the community health supply chain would be most impacted by improvements made in processes at the highest administrative strata, while exposing its delicate dependence on activities at the lowest levels. Visibility of inventory at all levels and improved data quality and use through a transparent tracking system have the potential to significantly reduce stockouts. Future interventions should take care to not develop parallel processes or exclude local health system agents to avoid disruption and ensure sustainable health outcome gains. Causal loop studies and system dynamics can further identify the systems interactions and relationships and their underlying causal mechanisms in need of intervention.

Improved data quality and statistical power of trial-level event-related potentials with Bayesian random-shift Gaussian processes

Studies of cognitive processes via electroencephalogram (EEG) recordings often analyze group-level event-related potentials (ERPs) averaged over multiple subjects and trials. This averaging procedure can obscure scientifically relevant variability across subjects and trials, but has been necessary due to the difficulties posed by inference of trial-level ERPs. We introduce the Bayesian Random Phase-Amplitude Gaussian Process (RPAGP) model, for inference of trial-level amplitude, latency, and ERP waveforms. We apply RPAGP to data from a study of ERP responses to emotionally arousing images. The model estimates of trial-specific signals are shown to greatly improve statistical power in detecting significant differences in experimental conditions compared to existing methods. Our results suggest that replacing the observed data with the de-noised RPAGP predictions can potentially improve the sensitivity and accuracy of many of the existing ERP analysis pipelines.

Retraction Note: Improvement of seismic data quality and recognition of fault discontinuities through seismic data conditioning applications: a case study of Issaran oil field, Gulf of Suez, Egypt

Retraction Note: Improvement of seismic data quality and recognition of fault discontinuities through seismic data conditioning applications: a case study of Issaran oil field, Gulf of Suez, Egypt

Particulate matter annual emission factors for dairy facilities and cattle feedlots of Texas, USA

The direct emission of inhalable particulate matters (PM2.5 and PM10) from agricultural sources imposes major concern to human health. It is imperative to keep regular update of the pollutant emissions to scrutinize the sources and take necessary preventive measures. Two representative animal feeding operations, one dairy and one feedlot of Texas were selected during the year 2021 for summer and winter sampling using programmable Federal Reference Method (FRM) samplers and Texas A&M University (TAMU) designed samplers. Additionally, for quick and simple emission concentration determination, handheld particle counters were utilized parallel to the FRM and TAMU designed samplers. The air quality index showed that the PM2.5 pollutant fell into ‘Moderate’ category for about 20.93% times and 13.89% times in the dairy and the feedlot respectively. The maximum mean of daily PM10 concentration observed in the Feedlot during the summer was 1676.3 μg/m3 which can be aggravating. The correlation developed using the emission data from the FRM and handheld samplers demonstrated linear relationship and 0.66 of goodness of fit. During the summer, maximum mean PM2.5 emission factors (EF) of 0.84 kg 1000 hd−1 d−1 and 8.09 kg 1000 hd−1 d−1 were determined for the dairy and for the feedlot respectively. However, the winter and the summer PM10 EF for the dairy did not differ significantly (P > 0.05). Although there has not been enough record of PM2.5 annual EF, the current study PM10 annual EF was benchmarked with some of the previously published EFs. This investigation on the air pollutant EFs from the animal feeding operations can greatly benefit in preventing the pollution and updating the most recent EFs. However, continuous monitoring and further research are suggested to improve data quality and develop strategic management practices.

Predictive health monitoring: Leveraging artificial intelligence for early detection of infectious diseases in nursing home residents through discontinuous vital signs analysis

This research addresses the problem of detecting acute respiratory, urinary tract, and other infectious diseases in elderly nursing home residents using machine learning algorithms. The study analyzes data extracted from multiple vital signs and other contextual information for diagnostic purposes. The daily data collection process encounters sampling constraints due to weekends, holidays, shift changes, staff turnover, and equipment breakdowns, resulting in numerous nulls, repeated readings, outliers, and meaningless values. The short time series generated also pose a challenge to analysis, preventing the extraction of seasonal information or consistent trends. Blind data collection results in most of the data coming from periods when residents are healthy, resulting in excessively imbalanced data. This study proposes a data cleaning process and then builds a mechanism that reproduces the basal activity of the residents to improve the classification of the disease. The results show that the proposed basal module-assisted machine learning techniques allow anticipating diagnostics 2, 3 or 4 days before doctors decide to start treatment with antibiotics, achieving a performance measured by the area-under-the-curve metric of 0.857. The contributions of this work are: (1) a new data cleaning process; (2) the analysis of contextual information to improve data quality; (3) the generation of a baseline measure for relative comparison; and (4) the use of either binary (disease/no disease) or multiclass classification, differentiating among types of infections and showing the advantages of multiclass versus binary classification. From a medical point of view, the anticipated detection of infectious diseases in institutionalized individuals is brand new.

An ORL-DLNN and IFIM-MST Framework for Data Quality Improvement in Modern Master Data Management

The process of generating and maintaining a master record for each person in a business is called Master Data Management (MDM). But, during Modern Master Data Management (MMDM), the prevailing methodologies did not focus on data loss. Hence, an Optimal with Recover Layer-based Deep Learning Neural Network (ORL-DLNN) and Indexed Fisher Information Matrix-based Minimum Spanning Tree-based Modern Master Data Management (IFIM-MST-based MMDM) is proposed in this paper. The source data is taken and pre-processed during training. Then, the products are clustered, and the attributes are extracted. Thereafter, by utilizing Dynamic Function-based Gold Rush Optimization (DF-GRO), the optimal attributes are selected. Afterward, for performing format classification with error detection, ORL-DLNN is introduced. Then, by utilizing the Reconciliation Rule (RR), the detected errors are corrected. After that, by utilizing the IFIM-MST, the text records are stored in the primary drive and the video and images are stored in the external drive. The details are entered by the information stewards into the classifier during testing. The data will be corrected if correction is present. Then, for verifying whether the data must be stored or not, some conditions are checked. Hence, the outcomes illustrated that the proposed system obtained a high accuracy of 98.3%, thus outperforming the existing techniques

TabFedSL: A Self-Supervised Approach to Labeling Tabular Data in Federated Learning Environments

Currently, self-supervised learning has shown effectiveness in solving data labeling issues. Its success mainly depends on having access to large, high-quality datasets with diverse features. It also relies on utilizing the spatial, temporal, and semantic structures present in the data. However, domains such as finance, healthcare, and insurance primarily utilize tabular data formats. This presents challenges for traditional data augmentation methods aimed at improving data quality. Furthermore, the privacy-sensitive nature of these domains complicates the acquisition of the extensive, high-quality datasets necessary for training effective self-supervised models. To tackle these challenges, our proposal introduces a novel framework that combines self-supervised learning with Federated Learning (FL). This approach aims to solve the problem of data-distributed training while ensuring training quality. Our framework improves upon the conventional self-supervised learning data augmentation paradigm by incorporating data labeling through the segmentation of data into subsets. Our framework adds noise by splitting subsets of data and can achieve the same level of centralized learning in a distributed environment. Moreover, we conduct experiments on various public tabular datasets to evaluate our approach. The experimental results showcase the effectiveness and generalizability of our proposed method in scenarios involving unlabeled data and distributed settings.

Concept of a Satellite Cross-Calibration Radiometer for In-Orbit Calibration of Commercial Optical Satellites

The satellite Earth observation (EO) sector is burgeoning with hundreds of commercial satellites being launched each year, delivering a rich source of data that could be exploited for societal benefit. Data streams from the growing number of commercial satellites are of variable quality, limiting the potential for their combined use in science applications that need long time-series data from multiple sources. The quality of calibration performed on optical sensors onboard many satellite systems is highly variable due to calibration methods, sensor design, mission objective, budget, or other operational constraints. A small number of currently operating well-characterised satellite systems with onboard calibration, such as Landsat-8/9 and Sentinel-2, and planned future missions, like the NASA Climate Absolute Radiance and Refractivity Observatory (CLARREO) Pathfinder, the European Space Agency (ESA)’s Traceable Radiometry Underpinning Terrestrial and Helio Studies (TRUTHS), and LIBRA from China, are considered benchmarks for optical data quality due to their traceability to international measurement standards. This paper describes the concept of a space-based transfer calibration radiometer called the Satellite Cross-Calibration Radiometer (SCR) that would enable the calibration parameters from satellites such as Landsat-8/9, Sentinel-2, or other benchmark systems to be transferred to a range of commercial optical EO satellite systems while in orbit. A description of the key characteristics of the SCR to successfully operate in orbit and transfer calibration from reference systems to client systems is presented. A system like the SCR in orbit could complement SI-Traceable satellites (SITSats) to improve data quality and consistency and facilitate the interoperable use of data from multiple optical sensor systems for delivering higher returns on the global investment in EO.

Standardizing Substrate Selection: A Strategy toward Unbiased Evaluation of Reaction Generality.

With over 10,000 new reaction protocols arising every year, only a handful of these procedures transition from academia to application. A major reason for this gap stems from the lack of comprehensive knowledge about a reaction's scope, i.e., to which substrates the protocol can or cannot be applied. Even though chemists invest substantial effort to assess the scope of new protocols, the resulting scope tables involve significant biases, reducing their expressiveness. Herein we report a standardized substrate selection strategy designed to mitigate these biases and evaluate the applicability, as well as the limits, of any chemical reaction. Unsupervised learning is utilized to map the chemical space of industrially relevant molecules. Subsequently, potential substrate candidates are projected onto this universal map, enabling the selection of a structurally diverse set of substrates with optimal relevance and coverage. By testing our methodology on different chemical reactions, we were able to demonstrate its effectiveness in finding general reactivity trends by using a few highly representative examples. The developed methodology empowers chemists to showcase the unbiased applicability of novel methodologies, facilitating their practical applications. We hope that this work will trigger interdisciplinary discussions about biases in synthetic chemistry, leading to improved data quality.

Current benefits and future possibilities with digital field reporting

Mobile phones and tablets enable contractors to digitally collect large amounts of production remarks and facilitate the acquisition. The increased data access and machine learning techniques allow the construction industry to take a significant step forward in shifting from implicit to explicit knowledge. However, this step requires both standardisation and data quality assurance combined with project incitements ensuring continuous data collection. Therefore, this study examines the current data quality and standardisation of inspection data generated using the production software Dalux Field, mining a dataset of more than 95000 production issues. Additionally, a survey of production software users assesses project and project member benefits and future possibilities with digital inspection reporting. The results show considerable benefits with digital inspection reporting, such as time savings, cost reductions and increased general quality control. However, the standardisation in reporting between projects and team members is low. Finally, this paper suggests methods for improving data quality and standardization for automation of the data analysis, allowing new projects in project-based organisations to benefit from previous project experiences.

A data mining-based framework to define optimal operation strategies for in-duct ultraviolet germicidal irradiation systems for different climatic conditions

This paper presents a novel data mining framework for optimizing the operation of in-duct Ultraviolet germicidal irradiation (UVGI) systems employed for air disinfection within air handling units. The performance of the UVGI system is influenced by the operating conditions of air handling units. As a result, under favourable operating conditions, the system produces excessive Ultraviolet C energy beyond the required disinfection levels, causing energy waste and accelerating the degradation of heating, ventilation, and air conditioning (HVAC) components. The framework aims to improve energy efficiency and mitigate material degradation within HVAC components by systematically adjusting UVGI system output based on changing operating conditions of air handling units throughout the year using advanced data mining techniques. The methodology involves several steps: Firstly, data pre-processing is performed to improve data quality. Then, the K-means clustering algorithm is applied to discover patterns in operational data. Subsequently, an effective performance index and baseline are defined to identify energy-saving opportunities within each cluster. Finally, the optimal operation strategy is defined based on the determined baseline. The framework is extended to five distinct climate zones to derive region-specific operation strategies. Results from a case study in a small office building in Montreal and an extension to five distinct climate zones in Canada demonstrate energy savings from 8% to 37%. This study offers a promising approach to optimize UVGI system operation, with potential applications in various energy systems in buildings requiring adaptive operational conditions for optimal performance.

Does it Pay to Send Multiple Pre-Paid Incentives? Evidence from a Randomized Experiment

To encourage survey participation and improve sample representativeness, the Survey of Consumer Finances (SCF) offers an unconditional pre-paid monetary incentive and separate post-paid incentive upon survey completion. We conducted a pre-registered between-subject randomized control experiment within the 2022 SCF, with at least 1,200 households per experimental group, to examine whether changing the pre-paid incentive structure affects survey outcomes. We assess the effects of: (1) altering the total dollar value of the pre-paid incentive (“incentive effect”), (2) giving two identical pre-paid incentives holding the total dollar value fixed (“reminder effect”), and (3) offering multiple pre-paid incentives of different amounts holding the total dollar value fixed (“slope effect”) on survey response rates, interviewer burden, and data quality. Our evidence indicates that a single $15 pre-paid incentive increases response rates and maintains similar levels of interviewer burden and data quality, relative to a single $5 pre-paid incentive. Splitting the $15 into two pre-paid incentives of different amounts increases interviewer burden though lengthening time in the field without improving response rates, reducing the number of contact attempts needed for a response, or improving data quality, regardless of whether the first pre-paid is larger or smaller than the second.

Dimensions of data sparseness and their effect on supply chain visibility

Supply chain visibility concerns the ability to track parts, components, or products in transit from supplier to customer. The data that organizations can obtain to establish or improve supply chain visibility is often sparse. This paper presents a classification of the dimensions of data sparseness and quantitatively explores the impact of these dimensions on supply chain visibility. Based on a review of supply chain visibility and data quality literature, this study proposes to characterize data sparseness as a lack of data quality across the entire supply chain, where data sparseness can be classified into three dimensions: noise, bias, and missing values. The quantitative analysis relies on a stylized simulation model of a moderately complex illicit supply chain. Scenarios are used to evaluate the combined effect of the individual dimensions from actors with different perspectives in the supply chain, either supply or demand-oriented. Results show that when a data sparseness of 90% is applied, supply chain visibility reduces to 52% for noise, to 65% for bias, and to 32% for missing values. The scenarios also show that companies with a supply-oriented view typically have a higher supply chain visibility than those with a demand-oriented view. The classification and assessment offer valuable insights for improving data quality and for enhancing supply chain visibility.

What do veterinary professionals and clients want? Using discrete choice methods in veterinary research to determine preferences.

Discrete choice methods (DCMs) are a suite of research techniques for identifying individual preferences using choice information. Widely utilized by other fields yet rarely employed in veterinary research, DCMs have tremendous potential to improve veterinary healthcare by understanding and incorporating owner and veterinary professionals' (encompassing veterinarians, veterinary clinicians, technicians, receptionists, attendants, etc) preferences to optimize the care continuum. DCMs have several advantages over other stated preference methods, such as ranking and ratings, including improved data quality and actionability. However, they are not a panacea, and limitations that may affect DCMs' application to the veterinary field are outlined alongside realistic mitigation strategies. The information provided aims to increase awareness of DCMs and their utility in veterinary research and encourage greater uptake as a more robust method for measuring preferences.

Enhancing Industrial IoT Security: Utilizing Blockchain-Assisted Deep Federated Learning for Collaborative Intrusion Detection

The Industrial Internet of Things (IIoT) is a rapidly evolving features with multiple applications, including critical infrastructure. Privacy policies are required to preserve the protection of user data in the threat intelligence community. Blockchain is a modern technology which used recently to provide more secure storage and efficiency. In this research, Blockchain Assisted Deep Federated Learning (BC_DFL) system is used to detect intruders. The three key processes used in the proposed intrusion detection architecture are data collection, pre-processing and intrusion detection. Data normalization, reduction, cleaning and transformation are used in pre-processing to remove extraneous information and improve data quality. This pre-processed data is sent to the Blockchain Assisted Deep Federated Learning (BC_DFL) system for intrusion detection. To detect intruders, the federated learning-based Capsule Auto-Encoder (FL_CAE) architecture first learns the properties from the inputs. Blockchain technology (BCTech) is not only used for storage but also improves security by eliminating the possibility of threatening node and individual server failure. The ToN_IoT and UNSW-NB15 data sets are used in the implementation and performance evaluation study. The proposed model is evaluated using existing in the Results section. In the UNSW-NB15 dataset, proposed model achieved an accuracy, precision, recall and F1 score of 97.26%, 97.28%, 96.89% and 96.96% respectively as well as in ToN_IoT data set proposed model achieved an accuracy, precision, recall and F1 score of 95.74%, 99.54 %, 99.49% and 99.24%, respectively. The execution of the proposed approach takes 2.31 seconds in the UNSW-NB15 dataset and 1.66 seconds in the ToN_IoT dataset. Blockchain offers a transparent and impenetrable ledger for transaction recording and verification. Within the framework of cooperative intrusion detection, it guarantees a secure and reliable exchange of threat intelligence and detection models between various users of IIoT ecosystem.

Analysis and Modelling of Structured Data with Automatic Data Analysis Web Application

Abstract: This paper introduces a novel web application designed for automated analysis and modelling of structured data. The application eliminates the need for user coding, allowing users to interact with the data through a user-friendly interface. By uploading a structured dataset, users can leverage the application's functionalities to improve data quality. These functionalities include, but are not limited to, data cleaning and pre-processing. One of the key strengths of this application is its ability to automatically generate machine learning models based on the uploaded data. Furthermore, the application takes the automation a step further by training the generated models without requiring user intervention. This eliminates the need for data science expertise, making the power of machine learning accessible to a wider audience. This paper details the application's architecture and functionalities, along with an evaluation of its effectiveness in data analysis and model building. The paper discusses the application's potential impact on democratizing data science and its limitations for future research directions.

Automating External Flat File Integration in SAP BW/4HANA: A Cost-Effective ABAP Approach

With SAP BW/4HANA as their enterprise data warehouse, organizations face challenges loading external data like CSV files into the system efficiently. Manual methods using GUIs are slow, error-prone, and lack scalability. This paper presents an automation solution using custom ABAP programming and standard BW/4HANA capabilities like process chains and advanced datastore objects (ADSOs) to orchestrate and load files. An ABAP program handles file discovery, logging, triggering process chains, monitoring status, and error handling. Benefits seen in production deployments include reduced load times and manual efforts, and improved data quality. While requiring upfront development effort, this approach provides a flexible, cost-effective data integration solution leveraging SAP's native tools. Key technical aspects include modular program design, use of SAP standard APIs, logging/status tracking, and BW process chain integration. Limitations are scalability for large volumes and complex transformations. The solution offers organizations an automation template for their BW/4HANA loading needs using in-house resources.

A trust and privacy-preserving intelligent big data collection scheme in mobile edge-cloud crowdsourcing

As one of important Edge-Cloud solutions, mobile crowd sensing (MCS) platform resides in the cloud, and recruits massive workers in the edge network to sense data, so big data can be collected to build various applications or services to consumers, which is a promising computing paradigm. There are two critical properties that are not well addressed. One is current privacy preserving (PP) scheme pays little attention to quality-ware based data collection, so malicious or low-trust workers can cause false or low-quality data attacks, which will seriously affect the foundation of data-based services. The second is the lack of an intelligent worker selection scheme so to reduce costs and improve data quality under PP condition. To address the above issues, in this paper, we first propose a big data collection framework that not only has the function of PP but also can effectively identify worker trust, which is not achieved by previous studies. In particular, the worker trust identification approach in this paper abandons the past unreasonable assumption that data quality can be identified by the obtained data of data requester (DR). Then, we propose a reinforcement-based worker selection scheme that comprehensively considers workers’ trust, workers’ quotation and workers’ locations, so as to overcome the problem of worker recruitment that cannot be quality-ware in the past location privacy protection. Theoretical analysis and massive experimental results demonstrate that the proposed strategy is better than the state-of-the-art strategies in data quality, system cost and other performance indicator

TB-HQ: An Incentive Mechanism for High-Quality Cooperation in Crowdsensing

Crowdsensing utilizes a range of sensing resources and participants, including mobile device sensors, to achieve collaborative sensing and information fusion. This enables it to handle complex social sensing tasks and provide more intelligent and real-time environment sensing services. Incentive mechanisms in crowdsensing are employed to address issues related to insufficient user participation and low-quality data submission. However, existing mechanisms fail to adequately consider reference points in user decision-making and uncertainty in the decision-making environment. This results in high incentive costs for the platform and limited effectiveness. On the one hand, the probabilities and utilities in the actual decision environment are defined based on user preferences, and uncertainty can lead to unpredictable impacts on users’ future gains or losses. On the other hand, users identify their choices based on certain known values, namely reference points. The factors influencing user decisions are not solely the absolute final result level but rather the relative changes or differences between the final result and the reference point. Therefore, to resolve this problem, we propose TB-HQ, an incentive mechanism for high-quality cooperation in crowdsensing, which simultaneously considers the reference points adopted by users in decision-making and the uncertainty caused by their preferences. This mechanism includes a task bonus-based incentive mechanism (TBIM) and a high quality-driven winner screening mechanism (HQWSM). TBIM motivates users to participate in tasks by offering task bonuses, which alter their reference points. HQWSM enhances data quality by reconstructing utility functions based on user preferences. Simulation results indicate that the proposed incentive mechanism is more effective in improving data quality and platform utility than the comparative incentive mechanisms, with a 32.7% increase in data quality and a 77.3% increase in platform utility.

Validation of Adult Sepsis Event and Epidemiologic Analysis of Sepsis Prevalence and Mortality Using Adult Sepsis Event's Electronic Health Records-Based Sequential Organ Failure Assessment Criteria: A Single-Center Study in South Korea.

In 2018, the Centers for Disease Control and Prevention introduced the Adult Sepsis Event (ASE) definition, using electronic health records (EHRs) data for surveillance and sepsis quality improvement. However, data regarding ASE outside the United States remain limited. We therefore aimed to validate the diagnostic accuracy of the ASE and to assess the prevalence and mortality of sepsis using ASE. Retrospective cohort study. A single center in South Korea, with 2732 beds including 221 ICU beds. During the validation phase, adult patients who were hospitalized or visiting the emergency department between November 5 and November 11, 2019, were included. In the subsequent phase of epidemiologic analysis, we included adult patients who were admitted from January to December 2020. None. ASE had a sensitivity of 91.6%, a specificity of 98.3%, a positive predictive value (PPV) of 57.4%, and a negative predictive value of 99.8% when compared with the Sepsis-3 definition. Of 126,998 adult patient hospitalizations in 2020, 6,872 cases were diagnosed with sepsis based on the ASE (5.4% per year), and 893 patients were identified as having sepsis according to the International Classification of Diseases , 10th Edition (ICD-10) (0.7% per year). Hospital mortality rates were 16.6% (ASE) and 23.5% (ICD-10-coded sepsis). Monthly sepsis prevalence and hospital mortality exhibited less variation when diagnosed using ASE compared with ICD-10 coding (coefficient of variation [CV] for sepsis prevalence: 0.051 vs. 0.163, Miller test p < 0.001; CV for hospital mortality: 0.087 vs. 0.261, p = 0.001). ASE demonstrated high sensitivity and a moderate PPV compared with the Sepsis-3 criteria in a Korean population. The prevalence of sepsis, as defined by ASE, was 5.4% per year and was similar to U.S. estimates. The prevalence of sepsis by ASE was eight times higher and exhibited less monthly variability compared with that based on the ICD-10 code.