Automated Data Quality Control Research Articles

Automated Data Quality Control System in Health and Demographic Surveillance System

The automated data quality management system serves as a comprehensive solution developed to enhance the precision, dependability, and uniformity of data within data-driven organizations. Such systems play an important role in eliminating the shortcomings associated with manual data quality management, which is prevalent in health and demographic surveillance systems (HDSS). The ongoing difficulty of ensuring data quality through manual processing hinders the HDSS's capacity to optimize data quality effectively. To address this challenge, our study adopted design science methodologies to provide guidelines for the design and implementation of the automated data quality control system. The open source technologies (Pentaho data integration, R Studio, SQL, Windows task scheduler) were used to facilitate the automation and validation of the incoming and database resident data. The quality of data has vastly improved since the implementation of the proposed system. The findings suggest that the automated data quality control system exhibited superior performance compared to the manual methods, thereby minimizing errors and time-wasting efforts.

Abstract LB079: An end-to-end Visium spatial transcriptomics computational pipeline for generating low-code interactive reports of spatial insights

Abstract In recent years, methods for analyzing and interpreting expression data have evolved rapidly. Newer and more complex datasets are being developed faster than ever, and the resources needed to perform rigorous data analysis have increased dramatically. In spatial transcriptomics analysis, researchers face many technical challenges: data management, compute power, and programming knowledge. In addition to these technical barriers, a constantly evolving field can make it difficult for scientists to understand if they are performing adequate quality control, proper data integration, choosing situationally correct methods, and showing accurate biological insights. Transcriptomics data comes with many meaningful caveats that make having the proper technical resources just a single piece of the puzzle. These obstacles keep scientists with valuable biological questions from maximizing the power of Spatial Transcriptomics. To address these challenges, we developed an end-to-end Visium spatial transcriptomics computational pipeline for generating interactive figures and reports for low-code exploration. This pipeline automated data quality control, batch-effects correction, and multiple peer-reviewed analysis techniques to perform a spatially resolved analysis of tissue heterogeneity. To demonstrate the effectiveness of this pipeline, we performed an analysis on a cohort consisting of 3 patients with adenocarcinoma and 1 patient with signet ring cell carcinoma, each with 1 sample of primary colon tumor and its matched adjacent normal tissue. All samples were formalin-fixed paraffin-embedded (FFPE) provided by BioChain Institute. The tissue sections were stained with Hematoxylin and Eosin, and the transcriptome was mapped using 10x Genomics Visium Spatial Gene Expression for FFPE. The raw base call files were then processed into feature-barcode matrices, followed by quality control measures that included data filtering, sample integration, batch-effect removal, and normalization. The pipeline then combined multiple analysis methods for feature extraction from genes and spatial spots, including clustering, neighborhood detection, and gene module detection. Further, we leveraged clinical metadata to perform differential expression and abundance analyses of the extracted features across different tissue and disease types. The pipeline finally generated an interactive report detailing analysis of spatially-resolved tissue biology, pathology, and microenvironment, allowing a scientist to analyze and explore the data further. This pipeline to identified differences in the spatial arrangement between tumor and normal tissue, in addition to demonstrating the spatial arrangements that differentiate adenocarcinoma and signet ring cell carcinoma relating to the invasive front and body of the tumors. Citation Format: Steven Hamel, Elim Cheung, Ying Qu, Malachi Loviska, Aaron Mayer, Lutong Zhang, Tong Lu, Vidyodhaya Sundaram, Baowen Zhang, Alex Trevino. An end-to-end Visium spatial transcriptomics computational pipeline for generating low-code interactive reports of spatial insights [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 2 (Clinical Trials and Late-Breaking Research); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(8_Suppl):Abstract nr LB079.

Toward Autonomous Detection of Anomalous GNSS Data Via Applied Unsupervised Artificial Intelligence

Artificial intelligence applications within the geosciences are becoming increasingly common, yet there are still many challenges involved in adapting established techniques to geoscience data sets. Applications in the realm of volcanic hazards assessment show great promise for addressing such challenges. Here, we describe a Jupyter Notebook we developed that ingests real-time Global Navigation Satellite System (GNSS) data streams from the EarthCube CHORDS (Cloud-Hosted Real-time Data Services for the geosciences) portal TZVOLCANO, applies unsupervised learning algorithms to perform automated data quality control (“noise reduction”), and explores autonomous detection of unusual volcanic activity using a neural network. The TZVOLCANO CHORDS portal streams real-time GNSS positioning data in 1[Formula: see text]s intervals from the TZVOLCANO network, which monitors the active volcano Ol Doinyo Lengai in Tanzania, through UNAVCO’s real-time GNSS data services. UNAVCO’s real-time data services provide near-real-time positions processed by the Trimble Pivot system. The positioning data (latitude, longitude and height) are imported into the Jupyter Notebook presented in this paper in user-defined time spans. The positioning data are then collected in sets by the Jupyter Notebook and processed to extract a useful calculated variable in preparation for the machine learning algorithms, of which we choose the vector magnitude for further processing. Unsupervised K-means and Gaussian Mixture machine learning algorithms are then utilized to locate and remove data points (“filter”) that are likely caused by noise and unrelated to volcanic signals. We find that both the K-means and Gaussian Mixture machine learning algorithms perform well at identifying regions of high noise within tested GNSS data sets. The filtered data are then used to train an artificial intelligence neural network that predicts volcanic deformation. Our Jupyter Notebook has promise to be used for detecting potentially hazardous volcanic activity in the form of rapid vertical or horizontal displacement of the Earth’s surface.

Automated Data Quality Control in FDOPA brain PET Imaging using Deep Learning

Introduction. With biomedical imaging research increasingly using large datasets, it becomes critical to find operator-free methods to quality control the data collected and the associated analysis. Attempts to use artificial intelligence (AI) to perform automated quality control (QC) for both single-site and multi-site datasets have been explored in some neuroimaging techniques (e.g. EEG or MRI), although these methods struggle to find replication in other domains. The aim of this study is to test the feasibility of an automated QC pipeline for brain [18F]-FDOPA PET imaging as a biomarker for the dopamine system.Methods. Two different Convolutional Neural Networks (CNNs) were used and combined to assess spatial misalignment to a standard template and the signal-to-noise ratio (SNR) relative to 200 static [18F]-FDOPA PET images that had been manually quality controlled from three different PET/CT scanners. The scans were combined with an additional 400 scans, in which misalignment (200 scans) and low SNR (200 scans) were simulated. A cross-validation was performed, where 80% of the data were used for training and 20% for validation. Two additional datasets of [18F]-FDOPA PET images (50 and 100 scans respectively with at least 80% of good quality images) were used for out-of-sample validation.Results. The CNN performance was excellent in the training dataset (accuracy for motion: 0.86 ± 0.01, accuracy for SNR: 0.69 ± 0.01), leading to 100% accurate QC classification when applied to the two out-of-sample datasets. Data dimensionality reduction affected the generalizability of the CNNs, especially when the classifiers were applied to the out-of-sample data from 3D to 1D datasets.Conclusions. This feasibility study shows that it is possible to perform automatic QC of [18F]-FDOPA PET imaging with CNNs. The approach has the potential to be extended to other PET tracers in both brain and non-brain applications, but it is dependent on the availability of large datasets necessary for the algorithm training.

China Kidney Disease Network (CK-NET) 2015 Annual Data Report

China Kidney Disease Network (CK-NET) 2015 Annual Data Report

The Pediatric Emergency Care Applied Research Network Registry: A Multicenter Electronic Health Record Registry of Pediatric Emergency Care.

Electronic health record (EHR)-based registries allow for robust data to be derived directly from the patient clinical record and can provide important information about processes of care delivery and patient health outcomes. A data dictionary, and subsequent data model, were developed describing EHR data sources to include all processes of care within the emergency department (ED). ED visit data were deidentified and XML files were created and submitted to a central data coordinating center for inclusion in the registry. Automated data quality control occurred prior to submission through an application created for this project. Data quality reports were created for manual data quality review. The Pediatric Emergency Care Applied Research Network (PECARN) Registry, representing four hospital systems and seven EDs, demonstrates that ED data from disparate health systems and EHR vendors can be harmonized for use in a single registry with a common data model. The current PECARN Registry represents data from 2,019,461 pediatric ED visits, 894,503 distinct patients, more than 12.5 million narrative reports, and 12,469,754 laboratory tests and continues to accrue data monthly. The Registry is a robust harmonized clinical registry that includes data from diverse patients, sites, and EHR vendors derived via data extraction, deidentification, and secure submission to a central data coordinating center. The data provided may be used for benchmarking, clinical quality improvement, and comparative effectiveness research.

Argo workstation: a key component of operational oceanography

Operational oceanography requires the quantity, quality, and availability of data set and the timeliness and effectiveness of data products. Without steady and strong operational system supporting, operational oceanography will never be proceeded far. In this paper we describe an integrated platform named Argo Workstation. It operates as a data processing and management system, capable of data collection, automatic data quality control, visualized data check, statistical data search and data service. After it is set up, Argo workstation provides global high quality Argo data to users every day timely and effectively. It has not only played a key role in operational oceanography but also set up an example for operational system.

Автоматизированный контроль качества данных в информационно-измерительных и управляющих системах MES-уровня

Автоматизированный контроль качества данных в информационно-измерительных и управляющих системах MES-уровня

Developmental validation of a custom panel including 273 SNPs for forensic application using Ion Torrent PGM

Utilizing massively parallel sequencing (MPS) technology for SNP testing in forensic genetics is becoming attractive because of the shortcomings of STR markers, such as their high mutation rates and disadvantages associated with the current PCR-CE method as well as its limitations regarding multiplex capabilities. MPS offers the potential to genotype hundreds to thousands of SNPs from multiple samples in a single experimental run. In this study, we designed a customized SNP panel that includes 273 forensically relevant identity SNPs chosen from SNPforID, IISNP, and the HapMap database as well as previously related studies and evaluated the levels of genotyping precision, sequence coverage, sensitivity and SNP performance using the Ion Torrent PGM. In a concordant study of the custom MPS-SNP panel, only four MPS callings were missing due to coverage reads that were too low (<20), whereas the others were fully concordant with Sanger's sequencing results across the two control samples, that is, 9947A and 9948. The analyses indicated a balanced coverage among the included loci, with the exception of the 16 SNPs that were used to detect an inconsistent allele balance and/or lower coverage reads among 50 tested individuals from the Chinese HAN population and the above controls. With the exception of the 16 poorly performing SNPs, the sequence coverage obtained was extensive for the bulk of the SNPs, and only three Y-SNPs (rs16980601, rs11096432, rs3900) showed a mean coverage below 1000. Analyses of the dilution series of control DNA 9948 yielded reproducible results down to 1ng of DNA input. In addition, we provide an analysis tool for automated data quality control and genotyping checks, and we conclude that the SNP targets are polymorphic and independent in the Chinese HAN population. In summary, the evaluation of the sensitivity, accuracy and genotyping performance provides strong support for the application of MPS technology in forensic SNP analysis, and the assay offers a straightforward sample-to-genotype workflow that could be beneficial in forensic casework with respect to both individual identification and complex kinship issues.

Automatic Data Quality Control of Observations in Wireless Sensor Network

Emerging multisource earth observation technologies represented by wireless sensor network (WSN) technology are widely used in land surface observation and simulation studies. Consequently, data quality control of massive observation data has brought challenges to researchers. This letter describes a comprehensive approach applied to automatic data quality control of WSN data. First, summarize the quality element of WSN observation data which can be achieved through automated methods by analyzing the characteristics of WSN observation data, and develop a decision algorithm for each quality element. Then, associate the data type and algorithm through data quality control rules. Finally, establish an automatic data quality control system based on data quality control rules. As a matter of fact, this system has run for one and a half years and processed more than 500 million observation data records without human intervention. Application results show that this method system can effectively control the data quality of WSN data automatically.

SGNoise - a tool for the ambient noise level analysis at superconducting gravimeter stations

The SGNoise represents a web tool for the near real-time analysis of data from superconducting gravimeters (SGs). Gravity residuals are computed on daily basis from raw SG data (1-s sampling rate) and analysed/visualized in time and frequency domain. It fulfils the main goal of the SGNoise — automatic data quality control of continuously operating geophysical instruments which provides a helpful service for operators and data users. The data quality quantification is represented by the evaluation of ambient noise level at SG stations by spectral analysis of gravity residuals and its visualization through spectrograms and probability density functions. Among others, it provides a possibility for comparison of noise levels at SG stations as demonstrated for Pecný, Wettzell and Strasbourg stations which are included in the SGNoise service at http://oko.asu.cas.cz/grav/. The SGNoise program package is written in PHP5 scripting language using the GD Graphics Library. Procedures used for data processing and analysis are consistent with previous works on noise level analysis at SG stations.

Simultaneous analysis of hundreds of Y-chromosomal SNPs for high-resolution paternal lineage classification using targeted semiconductor sequencing.

SNPs from the non-recombining part of the human Y chromosome (Y-SNPs) are informative to classify paternal lineages in forensic, genealogical, anthropological, and evolutionary studies. Although thousands of Y-SNPs were identified thus far, previous Y-SNP multiplex tools target only dozens of markers simultaneously, thereby restricting the provided Y-haplogroup resolution and limiting their applications. Here, we overcome this shortcoming by introducing a high-resolution multiplex tool for parallel genotyping-by-sequencing of 530 Y-SNPs using the Ion Torrent PGM platform, which allows classification of 432 worldwide Y haplogroups. Contrary to previous Y-SNP multiplex tools, our approach covers branches of the entire Y tree, thereby maximizing the paternal lineage classification obtainable. We used a default DNA input amount of 10ng per reaction but preliminary sensitivity testing revealed positive results from as little as 100pg input DNA. Furthermore, we demonstrate that sample pooling using barcodes is feasible, allowing increased throughput for lower per-sample costs. In addition to the wetlab protocol, we provide a software tool for automated data quality control and haplogroup classification. The unique combination of ultra-high marker density and high sensitivity achievable from low amounts of potentially degraded DNA makes this new multiplex tool suitable for a wide range of Y-chromosome applications.

Operational profiling of temperature using ground-based microwave radiometry at Payerne: prospects and challenges

Abstract. The motivation of this study is to verify theoretical expectations placed on ground-based microwave radiometer (MWR) techniques and to confirm whether they are suitable for supporting key missions of national weather services, such as timely and accurate weather advisories and warnings. We evaluate reliability and accuracy of atmospheric temperature profiles retrieved continuously by the microwave profiler system HATPRO (Humidity And Temperature PROfiler) operated at the aerological station of Payerne (MeteoSwiss) in the time period August 2006–December 2009. Assessment is performed by comparing temperatures from the radiometer against temperature measurements from a radiosonde accounting for a total of 2107 quality-controlled all-season cases. In the evaluated time period, the MWR delivered reliable temperature profiles in 86% of all-weather conditions on a temporal resolution of 12–13 min. Random differences between MWR and radiosonde are down to 0.5 K in the lower boundary layer and increase to 1.7 K at 4 km height. The differences observed between MWR and radiosonde in the lower boundary layer are similar to the differences observed between the radiosonde and another in-situ sensor located on a close-by 30 m tower. Temperature retrievals from above 4 km contain less than 5% of the total information content of the measurements, which makes clear that this technique is mainly suited for continuous observations in the boundary layer. Systematic temperature differences are also observed throughout the retrieved profile and can account for up to ±0.5 K. These errors are due to offsets in the measurements of the microwave radiances that have been corrected for in data post-processing and lead to nearly bias-free overall temperature retrievals. Different reasons for the radiance offsets are discussed, but cannot be unambiguously determined retrospectively. Monitoring and, if necessary, corrections for radiance offsets as well as a real-time rigorous automated data quality control are mandatory for microwave profiler systems that are designated for operational temperature profiling. In the analysis of a subset of different atmospheric situations, it is shown that lifted inversions and data quality during precipitation present the largest challenges for operational MWR temperature profiling.

Spatiotemporal Models for Data-Anomaly Detection in Dynamic Environmental Monitoring Campaigns

The ecological sciences have benefited greatly from recent advances in wireless sensor technologies. These technologies allow researchers to deploy networks of automated sensors, which can monitor a landscape at very fine temporal and spatial scales. However, these networks are subject to harsh conditions, which lead to malfunctions in individual sensors and failures in network communications. The resulting data streams often exhibit incorrect data measurements and missing values. Identifying and correcting these is time-consuming and error-prone. We present a method for real-time automated data quality control (QC) that exploits the spatial and temporal correlations in the data to distinguish sensor failures from valid observations. The model adapts to each deployment site by learning a Bayesian network structure that captures spatial relationships between sensors, and it extends the structure to a dynamic Bayesian network to incorporate temporal correlations. This model is able to flag faulty observations and predict the true values of the missing or corrupt readings. The performance of the model is evaluated on data collected by the SensorScope Project. The results show that the spatiotemporal model demonstrates clear advantages over models that include only temporal or only spatial correlations, and that the model is capable of accurately imputing corrupted values.

05/00044 Use of an automatic data quality control algorithm for crude oil viscosity data: Barrufet, M. A. and Dexheimer, D. Fluid Phase Equilibria, 2004, 219, (2), 113–121

05/00044 Use of an automatic data quality control algorithm for crude oil viscosity data: Barrufet, M. A. and Dexheimer, D. Fluid Phase Equilibria, 2004, 219, (2), 113–121

Use of an automatic data quality control algorithm for crude oil viscosity data

When building correlations for physical properties, the data used must be quality controlled to ensure suitable performance of regression procedures and provide acceptable predictions. When using data from different sources and large datasets, this screening could become tedious and laborious unless a systematic and automated consistency check is used. For this study, we had a database of almost 3000 records of PVT properties and black-oil viscosity data coming from 324 differential liberation tests performed in different commercial laboratories. We developed a procedure to “clean up” the data on a test basis, before processing it with a regression routine. We individually screened each test, identified outlying observations and removed those from the regression calculations. The criteria used to discard data relied on the numerical evaluation of the first derivative of selected functions of one variable. If the physical behavior is predicted correctly, these functions should be monotonic. For example oil viscosity (observed function) should always increase as the pressure in the differential liberation tests decreases. Forward and backward derivatives were used to account for the end points. The filtered data resulting from this quality control process consisted of 2324 observations. The data were used to adapt two commonly used compositional viscosity models, Pedersen’s and Lohrenz, Bray and Clark (LBC), such that these models can be used for black-oil systems when compositional data are missing. The oil viscosity ranged from 0.13 to 78.3 cP, with pressure from 14.7 to 5602 psia (0.1–38.62 MPa) and temperature from 537 to 766 R (298–425.55 K). The oil specific gravity ranged from 0.389 to 0.921. These models were validated against an independent dataset consisting of 150 observations. The two models had lower statistical errors than currently available correlations.

Development of a computer application for headache diagnosis: The headache diagnostic system

A computer-assisted decision support system for headache classification and diagnosis is presented. The system provides: structured headache data collection, automated data quality control and management, and automated headache classification. The system is based upon headache classification and diagnostic criteria recently developed by the International Headache Society. Development, logical structure and validation of the system are discussed. The system is shown to provide a general improvement in headache classification reliability and has utility for validation and improvement of present headache classification criteria.

昼光実測データの自動品質管理

昼光実測データの自動品質管理