Data Error Detection Research Articles

A REVIEW OF DIFFERENT MACHINE LEARNING ALGORITHMS IN DATA ERROR DETECTION AND CORRECTION PROBLEMS

A REVIEW OF DIFFERENT MACHINE LEARNING ALGORITHMS IN DATA ERROR DETECTION AND CORRECTION PROBLEMS

An Unsupervised Error Detection Methodology for Detecting Mislabels in Healthcare Analytics.

Medical datasets may be imbalanced and contain errors due to subjective test results and clinical variability. The poor quality of original data affects classification accuracy and reliability. Hence, detecting abnormal samples in the dataset can help clinicians make better decisions. In this study, we propose an unsupervised error detection method using patterns discovered by the Pattern Discovery and Disentanglement (PDD) model, developed in our earlier work. Applied to the large data, the eICU Collaborative Research Database for sepsis risk assessment, the proposed algorithm can effectively discover statistically significant association patterns, generate an interpretable knowledge base for interpretability, cluster samples in an unsupervised learning manner, and detect abnormal samples from the dataset. As shown in the experimental result, our method outperformed K-Means by 38% on the full dataset and 47% on the reduced dataset for unsupervised clustering. Multiple supervised classifiers improve accuracy by an average of 4% after removing abnormal samples by the proposed error detection approach. Therefore, the proposed algorithm provides a robust and practical solution for unsupervised clustering and error detection in healthcare data.

Anomaly-based error and intrusion detection in tabular data: No DNN outperforms tree-based classifiers

Recent years have seen a growing involvement of researchers and practitioners in crafting Deep Neural Networks (DNNs) that seem to outperform existing machine learning approaches for solving classification problems as anomaly-based error and intrusion detection. Undoubtedly, classifiers may be very diverse among themselves, and choosing one or another is typically due to the specific task and target system. Designing and training the optimal tabular data classifier requires extensive experimentation, sensitivity analyses, big datasets, and domain-specific knowledge that may not be available at will or considered a non-strategical asset by many companies and stakeholders. This paper compares, using a total of 23 public datasets: i) traditional (tree-based, statistical) supervised classifiers, ii) DNNs that are specifically designed for classifying tabular data, iii) DNNs for image classification that are applied to tabular data after converting data points into images, alone and as ensembles. Experimental results and related discussions show clear advantages in adopting tree-based classifiers for anomaly-based error and intrusion detection in tabular data as they outperform their competitors, including DNNs. Then, individual classifiers are compared against ensembles using different combinations of the classifiers considered in this study as base-learners, providing a unified final response through many meta-learning strategies. Results show that there is no benefit in building ensembles instead of using a tree-based classifier as Random Forests, eXtreme Gradient Boosting or Extra Trees. The paper concludes that anomaly-based error and intrusion detectors for critical systems should use the old (but gold) tree-based classifiers, which are also easier to fine-tune, and understand; plus, they require less time and resources to learn their model.

An improved approach to organising mobile edge computing in a 5G network

Mobile edge computing is an important element in ensuring the efficiency of the 5G network as a whole, as it enables data storage and computing at the network edge. Existing solutions do not fully address the issues of load distribution between computing nodes, and most solutions do not offer methods for verifying computations and controlling errors. Accordingly, this paper aims to develop an approach to the organization of mobile edge computing in a 5G mobile network that would authenticate distribution servers and computing nodes, manage the process of distributing computing nodes, have a procedure for verifying the correctness of calculations, and take into account the parameters of computing nodes during distribution. To achieve this goal, we propose to use the developed method. The method of load balancing and selection of computing nodes for edge computing via 5G allows for identifying available nodes and distributing computing blocks among them. It also provides mutual authentication of elements and includes a method of data verification and error detection for the MEC system. The provided solution allows for controlling errors during calculations and protecting the server from incorrect data. These methods are optimized according to minimum network resources and computing time criteria. These improvements increase the efficiency of mobile edge computing in a 5G network.

Vertical Attention-Based Siamese ConvLSTM Network for Argo Data Error Detection

Vertical Attention-Based Siamese ConvLSTM Network for Argo Data Error Detection

Efficient automated error detection in medical data using deep-learning and label-clustering

Medical datasets inherently contain errors from subjective or inaccurate test results, or from confounding biological complexities. It is difficult for medical experts to detect these elusive errors manually, due to lack of contextual information, limiting data privacy regulations, and the sheer scale of data to be reviewed. Current methods for training robust artificial intelligence (AI) models on data containing mislabeled examples generally fall into one of several categories—attempting to improve the robustness of the model architecture, the regularization techniques used, the loss function used during training, or selecting a subset of data that contains cleaner labels. This last category requires the ability to efficiently detect errors either prior to or during training, either relabeling them or removing them completely. More recent progress in error detection has focused on using multi-network learning to minimize deleterious effects of errors on training, however, using many neural networks to reach a consensus on which data should be removed can be computationally intensive and inefficient. In this work, a deep-learning based algorithm was used in conjunction with a label-clustering approach to automate error detection. For dataset with synthetic label flips added, these errors were identified with an accuracy of up to 85%, while requiring up to 93% less computing resources to complete compared to a previous model consensus approach developed previously. The resulting trained AI models exhibited greater training stability and up to a 45% improvement in accuracy, from 69 to over 99% compared to the consensus approach, at least 10% improvement on using noise-robust loss functions in a binary classification problem, and a 51% improvement for multi-class classification. These results indicate that practical, automated a priori detection of errors in medical data is possible, without human oversight.

The DetectDeviatingCells algorithm was a useful addition to the toolkit for cellwise error detection in observational data

ObjectivesWe evaluated the error detection performance of the DetectDeviatingCells (DDC) algorithm which flags data anomalies at observation (casewise) and variable (cellwise) level in continuous variables. We compared its performance to other approaches in a simulated dataset. Study Design and SettingWe simulated height and weight data for hypothetical individuals aged 2–20 years. We changed a proportion of height values according to predetermined error patterns. We applied the DDC algorithm and other error-detection approaches (descriptive statistics, plots, fixed-threshold rules, classic, and robust Mahalanobis distance) and we compared error detection performance with sensitivity, specificity, likelihood ratios, predictive values, and receiver operating characteristic (ROC) curves. ResultsAt our chosen thresholds error detection specificity was excellent across all scenarios for all methods and sensitivity was higher for multivariable and robust methods. The DDC algorithm performance was similar to other robust multivariable methods. Analysis of ROC curves suggested that all methods had comparable performance for gross errors (e.g., wrong measurement unit), but the DDC algorithm outperformed the others for more complex error patterns (e.g., transcription errors that are still plausible, although extreme). ConclusionsThe DDC algorithm has the potential to improve error detection processes for observational data.

Cross-source Data Error Detection Approach Based on Federated Learning

PDF HTML XML Export Cite reminder Cross-source Data Error Detection Approach Based on Federated Learning DOI: 10.21655/ijsi.1673-7288.00295 Author: Affiliation: Clc Number: Fund Project: Article | Figures | Metrics | Reference | Related | Cited by | Materials | Comments Abstract:With the emergence and accumulation of massive data, data governance has become an important manner to improve data quality and maximize data value. Specifically, data error detection is a crucial step to improve data quality, which has attracted wide attention from both industry and academia. At present, various detection methods tailored for a single data source have been proposed. However, in many real-world scenarios, data are not centrally stored or managed. Data from different sources but highly correlated can be employed to improve the accuracy of error detection. Unfortunately, due to privacy/security issues, cross-source data are often not allowed to be integrated centrally. To this end, this paper proposes FeLeDetect, a cross-source data error detection method based on federated learning, so as to improve the error detection accuracy by using cross-source data information on the premise of data privacy. First, a Graph-based Error Detection Model, namely GEDM, is presented to capture sufficient data features from each data source. On this basis, the paper then designs a federated co-training algorithm, namely FCTA, to collaboratively train GEDM by using different cross-source data without privacy leakage of data. Furthermore, the paper designs a series of optimization methods to reduce communication costs during federated learning and manual labeling efforts. Finally, extensive experiments on three real-world datasets demonstrate that (1) GEDM achieves an average improvement of 10.3% and 25.2% in terms of the $F1$ score in the local and centralized scenarios, respectively, outperforming all the five existing state-of-the-art methods for error detection; (2) the F1 score of the error detection by FeLeDetect is 23.2% on average higher than that by GEDM in the local scenario. Reference Related Cited by

DNA technology for big data storage and error detection solutions: Hamming code vs Cyclic Redundancy Check (CRC)

There is an increasing need for high-capacity, highdensity storage media that can retain data for a long time, due to the exponential development in the capacity of information generated. The durability and high information density of synthetic deoxyribonucleic acid (DNA) make it an attractive and promising medium for data storage. DNA data storage technology is expected to revolutionize data storage in the coming years, replacing various Big Data storage technologies. As a medium that addresses the need for high-latency, immutable information storage, DNA has several potential advantages. One of the key advantages of DNA storage is its extraordinary density. Theoretically, a gram of DNA can encode 455 exabytes, or 2 bits per nucleotide. Unlike other digital storage media, synthetic DNA enables large quantities of data to be stored in a biological medium. This reduces the need for traditional storage media such as hard disks, which consume energy and require materials such as plastic or metals, and also often leads to the generation of electronic waste when they become obsolete or damaged. Additionally, although DNA degrades over thousands of years under non-ideal conditions, it is generally readable. Furthermore, as DNA possesses natural reading and writing enzymes as part of its biological functions, it is expected to remain the standard for data retrieval in the foreseeable future. However, the high error rate poses a significant challenge for DNA-based information coding strategies. Currently, it is impossible to execute DNA strand synthesis, amplification, or sequencing errors-free. In order to utilize synthetic DNA as a storage medium for digital data, specialized systems and solutions for direct error detection and correction must be implemented. The goal of this paper is to introduce DNA storage technology, outline the benefits and added value of this approach, and present an experiment comparing the effectiveness of two error detection and correction codes (Hamming and CRC) used in the DNA data storage strategy.

Fundamental Results of Cyclic Codes over Octonion Integers and Their Decoding Algorithm

Coding theory is the study of the properties of codes and their respective fitness for specific applications. Codes are used for data compression, cryptography, error detection, error correction, data transmission, and data storage. Codes are studied by various scientific disciplines, such as information theory, electrical engineering, mathematics, linguistics, and computer science, to design efficient and reliable data transmission methods. Many authors in the previous literature have discussed codes over finite fields, Gaussian integers, quaternion integers, etc. In this article, the author defines octonion integers, fundamental theorems related to octonion integers, encoding, and decoding of cyclic codes over the residue class of octonion integers with respect to the octonion Mannheim weight one. The comparison of primes, lengths, cardinality, dimension, and code rate with respect to Quaternion Integers and Octonion Integers will be discussed.

IoT data cleaning techniques: A survey

Data cleaning is considered as an effective approach of improving data quality in order to help practitioners and researchers be devoted to downstream analysis and decision-making without worrying about data trustworthiness. This paper provides a systematic summary of the two main stages of data cleaning for Internet of Things (IoT) data with time series characteristics, including error data detection and data repairing. In respect to error data detection techniques, it categorizes an overview of quantitative data error detection methods for detecting single-point errors, continuous errors, and multidimensional time series data errors and qualitative data error detection methods for detecting rule-violating errors. Besides, it provides a detailed description of error data repairing techniques, involving statistics-based repairing, rule-based repairing, and human-involved repairing. We review the strengths and the limitations of the current data cleaning techniques under IoT data applications and conclude with an outlook on the future of IoT data cleaning.

IoT-Based Harmful Toxic Gases Monitoring and Fault Detection on the Sensor Dataset Using Deep Learning Techniques

One of the main reasons for accidents among workers is harmful gas leakage. Many people die in chemical industries and their surrounding areas. The present invention is responsible for monitoring and controlling hazardous toxic gases like nitrogen dioxide (NO2), carbon monoxide, ozone (O3), sulfur dioxide (SO2), LPG, hydrocarbon gases, silicones, hydrocarbons, alcohol, CH4, hexane, benzine, as well as environmental conditions, such as temperature and relative humidity to prevent industrial accidents. The Arduino UNO R3 board is used as the central microcontroller. It is connected to the Cloud via AQ3 sensor, Minipid 2 HS PID sensor, IR5500 open path infrared gas detector, DHT11 Temperature and Humidity Sensor, MQ3 sensor, and ESP8266 and WIFI Module, which can store real-time sensor data and send alert messages to the industry’s safety control board. Machine learning and artificial intelligence will be used to make an intelligent prediction (AI). The information gathered will be examined in real-time. The real-time data provided through the sensor can be accessed worldwide. Sensor data quality is critical in the Internet of Things (IoT) applications because poor data quality renders them useless. Error detection in sensor data improves the IoT-based toxic gas monitoring, controlling, and prediction system. Live data from sensors or datasets should be analyzed properly using appropriate techniques. Hence, hybrid hidden Markov and artificial intelligence models are applied as an error detection technique in the sensor dataset. This technique outperformed the dataset gas sensor array under dynamic gas mixtures and lived data. Our method outperformed harmful gas monitoring and error detection in sensor datasets compared to other existing technologies. The hybrid HMM and ANN fault detection methods performed well on the datasets and produced 0.01% false positive rate.

Perancangan dan Implementasi Encoder dan Decoder CRC-8 untuk Pendeteksian Error pada Transmisi Data antar Perangkat IoT

Cyclic Redundancy Check (CRC) is a widely used error detection code in digital communication systems to detect errors in data transmitted between transmitter and receiver. With the development of Internet-of-Things (IoT) technology, where many devices communicate wirelessly, the application of error detection codes such as CRC is essential to obtain reliable communication. The computation of CRC can be implemented in hardware using a linear feedback shift register (LFSR) or in software with shift and bitwise exclusive OR (XOR) operations for polynomial division. Due to the frequent use of CRC in wireless communication for IoT, research for the design and implementation of CRC continues to this day. This study aims to design and implement the encoder and decoder of CRC-8 with generator polynomial 0x07 on IoT devices. Two algorithms for computing the CRC, namely bitwise computation using LFSR and another one using lookup tables, are implemented on the Arduino Uno R3 board. Two Arduino boards connected in serial were used to verify the encoder and decoder implementation in the data transmission and error detection process. The experiment results showed that the encoder could correctly calculate the CRC value from the input data. Furthermore, the computation using lookup tables takes about four times faster than the bitwise method but requires more memory. In contrast, the bitwise computation method requires less memory but slower computation time. When testing using input data of 128 characters, the bitwise-based encoder requires a computation time of 2.37 milliseconds, while the lookup table-based encoder utilizing a lookup table requires a computation time of 0.5 milliseconds. In the error detection test, the results demonstrated that the receiver could detect errors in the transmitted data with a percentage of 100%.

Restricted Hamming–Huffman trees

We study a special case of Hamming–Huffman trees, in which both data compression and data error detection are tackled on the same structure. Given a hypercubeQnof dimensionn, we are interested in some aspects of its vertex neighborhoods. For a subsetLof vertices ofQn, the neighborhood ofLis defined as the union of the neighborhoods of the vertices ofL. The minimum neighborhood problem is that of determining the minimum neighborhood cardinality over all those setsL. This is a well-known problem that has already been solved. Our interest lies in determining optimal Hamming–Huffman trees, a problem that remains open and which is related to minimum neighborhoods inQn. In this work, we consider a restricted version of Hamming–Huffman trees, called [k]-HHT s, which admit symbol leaves in at mostkdifferent levels. We present an algorithm to build optimal [2]-HHT s. For uniform frequencies, we prove that an optimal HHT is always a [5]-HHT and that there exists an optimal HHT which is a [4]-HHT . Also, considering experimental results, we conjecture that there exists an optimal tree which is a [3]-HHT .

Enhancement of safety communication model

Abstract A fundamental measure of safety communication is the residual error probability, i. e., the probability of undetected errors. For the detection of data errors, typically a Cyclic Redundancy Check (CRC) is applied, and the resulting residual error probability is determined based on the Binary Symmetric Channel (BSC) model. The use of this model had been questioned since several error types cannot be sufficiently described. Especially the increasing introduction of security algorithms into underlying communication layers requires a more adequate channel model. This paper introduces an enhanced model that extends the list of considered data error types by combining the BSC model with a Uniformly Distributed Segments (UDS) model. Although models beyond BSC are applied, the hitherto method of the calculation of the residual error probability can be maintained.

There is no way! Ternary qualitative spatial reasoning for error detection in map data

AbstractDetection and correction of errors in map data based on spatial reasoning may be used to improve their quality. However, the majority of current spatial reasoning approaches are based on binary spatial relations and are not able to perform analyses involving more than two objects. This article proposes building accessibility analysis with the ternary ray intersection model to detect potential map errors. Where buildings are not accessible from the road network, this may indicate potential errors in map data such as roads that are not mapped. The plausibility of the proposed method was tested in a case study on OpenStreetMap data. The results have been published in an online mapping challenge where volunteering mappers have used them to correct errors in map data, and have provided feedback on the analysis. The results show that the proposed method can detect errors in map data that are caused by incorrect classification of buildings, incorrect mapping of multi‐part buildings, and missing road data.

269: Educational Adaptation for the COVID-19 Pandemic: A Virtual Model Enables Ongoing Training

Jordan, Mary1; Evans, Susan2; Spangler, Lacey1; Miles, William3; Brintzenhoff, Rita3 Author Information

A Rule-Based Data Quality Assessment System for Electronic Health Record Data.

Rule-based data quality assessment in health care facilities was explored through compilation, implementation, and evaluation of 63,397 data quality rules in a single-center case study to assess the ability of rules-based data quality assessment to identify data errors of importance to physicians and system owners. We applied a design science framework to design, demonstrate, test, and evaluate a scalable framework with which data quality rules can be managed and used in health care facilities for data quality assessment and monitoring. We identified 63,397 rules partitioned into 28 logic templates. A total of 819,683 discrepancies were identified by 4.5% of the rules. Nine out of 11 participating clinical and operational leaders indicated that the rules identified data quality problems and articulated next steps that they wanted to take based on the reported information. The combined rule template and knowledge table approach makes governance and maintenance of otherwise large rule sets manageable. Identified challenges to rule-based data quality monitoring included the lack of curated and maintained knowledge sources relevant to data error detection and lack of organizational resources to support clinical and operational leaders with investigation and characterization of data errors and pursuit of corrective and preventative actions. Limitations of our study included implementation within a single center and dependence of the results on the implemented rule set. This study demonstrates a scalable framework (up to 63,397 rules) with which data quality rules can be implemented and managed in health care facilities to identify data errors. The data quality problems identified at the implementation site were important enough to prompt action requests from clinical and operational leaders.

Isothermal Chemical Denaturation: Data Analysis, Error Detection, and Correction by PARAFAC2.

Characterization of a protein's conformational stability is a key step in the development of biotherapeutics, where protein unfolding leads to adverse properties, such as aggregation and loss of efficacy. Isothermal chemical denaturation (ICD) can be applied to determine chemical stability, aiming to identify the optimal solvent conditions, in terms of pH, salt concentration, and added excipients. For seven monoclonal antibodies, this study investigates the observed intrinsic protein fluorescence emission spectra as a function of denaturant concentration. Protein formulations are screened in two experimental series. We show how the peak shapes of folded and unfolded proteins are preserved under added salt (0-140 mM NaCl) and added excipients concentrations, as typically found in biotherapeutic formulations and that only minor effects in tryptophan fluorescence peak tailing are observed over a large pH range (5.5-9.0). The data of seven mAbs, where GuHCl was a suitable denaturant, are modeled using PARAFAC2. PARAFAC2, a linear decomposition method, is well suited for the data and yields robust, valid, and automated models that allow for the detection of erroneous measurements. Analysis of the errors show correlation with the well-based experimental setup, and differences in observed errors between the two experimental series. We additionally show a correction method for these outliers based on PARAFAC2 model scores, such that full transition curves can be retrieved, increasing the accuracy of any subsequent analysis.

Framework for Error Detection & its Localization in Sensor Data Stream for reliable big sensor data analytics using Apache Spark Streaming

Internet of Things (IoT) is one of the big sources of Big Sensor data in which collected sensor readings often polluted due to corruption and losses of the sensor readings. In Big Sensor data analytics, fast and efficient detection of error and its localization is a challenging research issue. Most of the existing solutions for the error detection and localization in sensor data use offline techniques for detecting errors. In this paper, we propose a novel framework for online error detection and its localization which uses an online scheme for detecting and localizing errors in sensor data using latest big data processing tools such as Apache Spark streaming. Performance evaluation of the proposed framework is done using two different datasets such as real-time air quality dataset of the Raipur city and Intel sensor dataset in terms of false positive.