Problem Of Error Detection Research Articles

A REVIEW OF DIFFERENT MACHINE LEARNING ALGORITHMS IN DATA ERROR DETECTION AND CORRECTION PROBLEMS

A REVIEW OF DIFFERENT MACHINE LEARNING ALGORITHMS IN DATA ERROR DETECTION AND CORRECTION PROBLEMS

Sparse Embedded Convolution Based Dual Feature Aggregation 3D Object Detection Network

The algorithm design of compatible detection speed and accuracy based on LiDAR point clouds is a challenging issue in various practical applications of 3D object detection, including the field of autonomous driving. This paper designs a single-stage object detection algorithm that is lightweight and compatible with detection speed and accuracy for the above issue. To achieve these objectives, we propose a framework for a 3D object detection algorithm using a single-stage detection network as the backbone network. Firstly, we design a dual feature extraction module to reduce the occurrence of vehicle miss and error detection problems. Then, we use a multi-scale feature fusion scheme to fuse feature information with different scales. Furthermore, we design a data enhancement scheme suitable for this network architecture. Experimental results in the KITTI dataset show that the proposed method achieves improvement ratios of 38.5% for the detection speed and 2.88% ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document} 13.65% in terms of the average precision of vehicle detection compared to the existing algorithm based on single-stage object detection (SECOND).

Automatic meter error detection with a data-driven approach

Meter error is one of the main contributing factors to unexpected fuel losses or gains in storage tanks at service stations. Although fuel dispensers are expected to be calibrated to standard accuracy periodically to ensure fair and reliable trade in the fuel market, some fuel retailers are unable to keep up with the standards. The current industry practice relies on onsite inspection to identify the issue, which leads to a cost burden if inspections are scheduled too frequently. To the best of our knowledge, there is no previously reported research tailored to the remote meter error detection problem. In this paper, we propose a novel framework for remote and automatic meter error detection via a data-driven approach based on inventory data and fuel transaction data. Specifically, we propose to use mean shift change point detection methods, including statistical-based as well as deep learning-based methods (LSTM-VAE, VAE, Kernel learning), to approach the problem. We present results on our data sets containing both real-world and simulated meter error data, and further evaluate these methods on several widely-used benchmark datasets, to assess their validity, advantages and limitations. The obtained results show that LSTM-VAE outperforms other models in most of the settings for the meter error dataset and the benchmark datasets.

A comparative study of anomaly detection methods for gross error detection problems

The chemical industry requires highly accurate and reliable measurements to ensure smooth operation and effective monitoring of processing facilities. However, measured data inevitably contains errors from various sources. Traditionally in flow systems, data reconciliation through mass balancing is applied to reduce error by estimating balanced flows. However, this approach can only handle random errors. For non-random errors (called gross errors, GEs) which are caused by measurement bias, instrument failures, or process leaks, among others, this approach would return incorrect results. In recent years, many gross error detection (GED) methods have been proposed by the research community. It is recognised that the basic principle of GED is a special case of the detection of outliers (or anomalies) in data analytics. With the developments of Machine Learning (ML) research, patterns in the data can be discovered to provide effective detection of anomalous instances. In this paper, we present a comprehensive study of the application of ML-based Anomaly Detection methods (ADMs) in the GED context on a number of synthetic datasets and compare the results with several established GED approaches. We also perform data transformation on the measurement data and compare its associated results to the original results, as well as investigate the effects of training size on the detection performance. One class Support Vector Machine outperformed other ADMs and five selected statistical tests for GED on Accuracy, F1 Score, and Overall Power while Interquartile Range (IQR) method obtained the best selectivity outcome among the top 6 AMDs and the five statistical tests. The results indicate that ADMs can potentially be applied to GED problems.

Fabrication progress detection for concrete T-girders based on improved Yolov4

Large precast concrete girder plants have many processes, long cycles and a large amount of data. This study proposes an improved Yolov4 object detection algorithm with a spatio-temporal relationship to detect each fabrication process of precast concrete girders. It realises the digitisation of the fabrication information of traditional precast concrete girder plants. Initially, adding upsampling and convolution layers to the Yolov4 base model enhances the feature extraction ability of the algorithm at different fabrication stages of precast concrete girders. The spatio-temporal relationship is adopted to determine the fabrication progress of precast concrete girders with identical features but are at various fabrication stages. Finally, this research conducts an application analysis of an actual precast concrete girder plant. The analysis result indicated that the improved Yolov4 algorithm significantly raises the mean average precision and average intersection over union in recognition. Besides, the spatio-temporal relationship effectively solves error detection problems caused by the similar appearance at different fabrication stages. This method provides practical support for digitising the fabrication data of traditional precast girder plants.

A Light-Weight Neural Network Using Multiscale Hybrid Attention for Building Change Detection

The study of high-precision building change detection is essential for the sustainable development of land resources. However, remote sensing imaging illumination variation and alignment errors have a large impact on the accuracy of building change detection. A novel lightweight Siamese neural network building change detection model is proposed for the error detection problem caused by non-real changes in high-resolution remote sensing images. The lightweight feature extraction module in the model acquires local contextual information at different scales, allowing it to fully learn local and global features. The hybrid attention module consisting of the channel and spatial attention can make full use of the rich spatiotemporal semantic information around the building to achieve accurate extraction of changing buildings. For the problems of large span of changing building scales, which easily lead to rough extraction of building edge details and missed detection of small-scale buildings, the multi-scale concept is introduced to divide the extracted feature maps into multiple sub-regions and introduce the hybrid attention module separately, and finally, the output features of different scales are weighted and fused to enhance the edge detail extraction capability. The model was experimented on the WHU-CD and LEVIR-CD public data sets and achieved F1 scores of 87.8% and 88.1%, respectively, which have higher change detection accuracy than the six comparison models, and only cost 9.15 G MACs and 3.20 M parameters. The results show that our model can achieve higher accuracy while significantly reducing the number of model parameters.

Combining GCN and Transformer for Chinese Grammatical Error Detection

<p>This paper describes our system at a task: Chinese Grammatical Error Diagnosis (CGED). The task is held by the Natural Language Processing Techniques for Educational Applications (NLP-TEA) to encourage the development of automatic grammatical error diagnosis in Chinese learning since 2014. The goal of CGED is to diagnose four types of grammatical errors: word selection (S), redundant words (R), missing words (M), and disordered words (W). The automatic CGED system contains two parts including error detection and error correction and our system is designed to solve the error detection problem. Our system is built on three models: 1) a BERT-based model leveraging syntactic information; 2) a BERT-based model leveraging contextual embeddings; 3) a lexicon-based graph neural network leveraging lexical information. We also design an ensemble mechanism to improve the single model’s performance. Finally, our system achieves the highest F1 scores at detection level and identification level among all teams participating in the CGED 2020 task.</p> <p> </p>

An extended human reliability analysing under fuzzy logic environment for ship navigation

ABSTRACT Preparation for a sea voyage is one of the fundamental aspects of navigation. Several complexities are involved during the preparation of the ship for navigation due to the nature of maritime work. At this point, analysing human-related error is of paramount importance to ensure the safety of the ship and the crew. This paper describes the principles of a methodology, namely fuzzy-based shipboard operation human reliability analysis (SOHRA), to quantitatively perform human error assessment through procedures of preparing the ship for navigation. While the SOHRA (a marine-specific HRA approach) quantifies human error, the fuzzy logic deals with ambiguity and vagueness in the human error detection problem. The findings show that the total HEP (Human error probability) is found 1.49E-01 for preparing the ship for navigation. Consequently, the paper provides practical contributions to shore-based safety professionals, ship managers, and masters of the ship since it performs a systematic human reliability assessment and enhances safety control levels in the operational aspect.

High-precision co-phase method for segments based on a convolutional neural network

In order to achieve the resolution comparable to the resolution of a monolithic primary mirror telescope and make the imaging quality of the imaging system reach or approach to the diffraction limit, the submirrors of the segments telescope should ensure co-phase splicing. To solve the problem of phase error detection, a high-precision piston error detection method is proposed based on convolutional neural network (CNN). By setting a mask with a sparse multi-subpupil configuration on the exit pupil of the imaging system, a point spread function (PSF) image dataset that is extremely sensitive to the piston error is constructed. According to the characteristics of this dataset, a high-performance CNN model is built. And the best detection range of CNN is tested. The simulation results show that a single network can accurately output the piston error of one or more submirrors in the capture range slightly less than one wavelength. When the single network is applied to the six-submirror imaging system, the detection precision of the piston error reaches an RMS value of 0.0013<i>λ</i> (here, RMS stands for root mean square). And the method has good robustness to residual tip-tilt error, wavefront aberration, and CCD noise, light source bandwidth. The method is simple and fast, and can be widely used to detect the piston error of the segments.

Unmanned aerial vehicle reconnaissance image recognition based on convolutional neural network

The important strategic position of military UAVs and the wide application of civil UAVs in many fields, they all mark the arrival of the era of unmanned aerial vehicles. At present, in the field of image research, recognition and real-time tracking of specific objects in images has been a technology that many scholars continue to study in depth and need to be further tackled. Image recognition and real-time tracking technology has been widely used in UAV aerial photography. Through the analysis of convolution neural network algorithm and the comparison of image recognition technology, the convolution neural network algorithm is improved to improve the image recognition effect. In this paper, a target detection technique based on improved Faster R-CNN is proposed. The algorithm model is implemented and the classification accuracy is improved through Faster R-CNN network optimization. Aiming at the problem of small target error detection and scale difference in aerial data sets, this paper designs the network structure of RPN and the optimization scheme of related algorithms. The structure of Faster R-CNN is adjusted by improving the embedding of CNN and OHEM algorithm, the accuracy of small target and multitarget detection is improved as a whole. The experimental results show that: compared with LENET-5, the recognition accuracy of the proposed algorithm is significantly improved. And with the increase of the number of samples, the accuracy of this algorithm is 98.9%.

Networked sensor data error estimation

Nowadays, the effectiveness of any smart transportation management or control strategy would heavily depend on reliable traffic data collected by sensors. Two problems regarding sensor data quality have received attention: first, the problem of identifying malfunctioning sensors; second, reconstruction of traffic flow. Most existing studies concerned about identifying completely malfunctioning sensors whose data should be discarded. In this paper, we focus on the problem of error detection and data recovery of partially malfunctioning sensors that could provide valuable information. By integrating a sensor measurement error model and a transportation network model, we propose a Generalized Method of Moments (GMM) based estimation approach to determine the parameters of systematic and random errors of traffic sensors in a road network. The proposed method allows flexible data aggregation that ameliorates identification and accuracy. The estimates regarding both systematic and random errors are utilized to conduct hypothesis test on sensor health and to estimate true traffic flows with observed counts. The results of three network examples with different scales demonstrate the applicability of the proposed method in a large variety of scenarios.

Автоматизация обнаружения и анализа ошибок в гиперконвергентных системах

The paper is devoted to the problem of early error detection and analysis in hyperconverged systems. One approach to organizing hyperconverged systems is to install on each physical server a separate instance of an operating system (OS) that carries virtualization tools and tools for administering and using a distributed data warehouse. Errors can occur both at the level of a single OS instance and at the level of the entire cluster. For example, incorrect control element commands from one infrastructure node can cause software failure on another node. In addition, errors from the subsystems of the cluster can provoke abnormal situations inside virtual machines. The complexity of the architecture of hyperconverged systems makes it difficult to analyze the errors that occur in them. To simplify such an analysis and increase its effectiveness, it is necessary to automate the process of detecting problems and collecting data necessary for their study and correction. Existing approaches for automation of error detection are described and various improvements are suggested to adopt them for systems where distributed storage and virtualization technologies are actively used. Improvements include log collection from the whole cluster just after the error occurred, additional analysis of guest operating system behaviour inside virtual machines, usage of a knowledge base for automated crash recovery and duplicate detection. Finally, a real-life scenario of error handling process in Virtuozzo company products is described starting from error detection and ending with fix deployment.

A Novel Vehicle Detection Method Based on the Fusion of Radio Received Signal Strength and Geomagnetism.

A geomagnetic signal blind zone exists between the front and rear axle of high-chassis vehicle such as trucks and buses, which leads to multiple-detection problem in detecting those vehicles running at low speed on roads or error-detection problem in the case of the stopping position of the vehicle is not standard when waiting for the traffic light to change. In order to improve the detection accuracy of any type of vehicle running at any speed, a novel two-sensors data fusion vehicle detection method through combining received signal strength from radio stations with geomagnetism around vehicles is designed and verified in the paper. Experimental results show that the accuracy of our proposed method can reach 95.4% and traditional single magnetism-based detection method was only 83.4% in the detection of high-chassis vehicles.

Automated Detection of Fetal Movements Based on Baseline Calculation and Fetal Heart Rate Accelerations

Fetal electronic monitoring is extensive and important in obstetrics. Although fetal movement is ususally used as an important indicator for quantifying fetal wellbeing, non-invasive and long-term monitoring of fetal movement remains challenging. The object of this study is to develop an algorithm for automatic detection of the fetal movements based on the analysis of Doppler ultrasound signals. In order to detect fetal movements automatically, a two-step process was proposed to track fetal movement. In Step 1, to suppress the problem of error detection, we calculated the baseline of the fetal movement signals from actography to extract new signals. In step2, we recalculated the threshold value of fetal movement detection by utilizing the information of fetal heart rate (FHR) acceleration to produced adaptive threshold values. The results showed that the union of results detected by the proposed method from actography and tocography achieved an encouraging performance with highest sensitivity and acceptable positive predictive value (PPV).

On Error Detection in Asymmetric Channels

We study the error detection problem in $ q $-ary asymmetric channels wherein every input symbol $ x_i $ is mapped to an output symbol $ y_i $ satisfying $ y_i \geq x_i $. A general setting is assumed where the noise vectors are (potentially) restricted in: 1) the amplitude, $ y_i - x_i \leq a $, 2) the Hamming weight, $ \sum_{i=1}^n 1_{\{y_i \neq x_i\}} \leq h $, and 3) the total weight, $ \sum_{i=1}^n (y_i - x_i) \leq t $. Optimal codes detecting these types of errors are described for certain sets of parameters $ a, h, t $, both in the standard and in the cyclic ($ \operatorname{mod}\, q $) version of the problem. It is also demonstrated that these codes are optimal in the large alphabet limit for every $ a, h, t $ and every block-length $ n $.

Can we trust Big Data? Applying philosophy of science to software

We address some of the epistemological challenges highlighted by the Critical Data Studies literature by reference to some of the key debates in the philosophy of science concerning computational modeling and simulation. We provide a brief overview of these debates focusing particularly on what Paul Humphreys calls epistemic opacity. We argue that debates in Critical Data Studies and philosophy of science have neglected the problem of error management and error detection. This is an especially important feature of the epistemology of Big Data. In “Error” section we explain the main characteristics of error detection and correction along with the relationship between error and path complexity in software. In this section we provide an overview of conventional statistical methods for error detection and review their limitations when faced with the high degree of conditionality inherent to modern software systems.

Distributed algorithms for convexified bad data and topology error detection and identification problems

Large-scale smart grids call for online algorithms that are able to achieve the most accurate estimates. This paper shows how to achieve both the scalability and near globally optimal results for bad data and topology error detection and identification problems, by conducting fully distributed algorithms over convexified problem formulations. The proposed distributed decomposition is realized by (1) reducing a large network into much smaller network “cliques” which do not need extensive information exchange; (2) performing a Lagrangian dual decomposition in each clique and passing messages between cliques; and (3) conducting alternative coordinate descent optimization for robustness. To reduce the relaxation error in the convexification procedure, a nuclear norm penalty is added to approximate original problems. Finally, we propose a new metric to evaluate detection and identification results, which enables a system operator to characterize confidence for further system operations. We show that the proposed algorithms can be realized on IEEE test systems with improved accuracy in a short time.

Comparing Drools and Ontology Reasoning Approaches for Automated Monitoring in Telecommunication Processes

Automated reconfiguration is one of the crucial tasks in telecommunication service composition. The first step in reconfiguration is the monitoring phase. The problem of monitoring and error detection frequently appears in different telecommunications architectures. This article describes the main components of the architecture for monitoring module in AUTO framework. The monitoring approach is based on semantic technologies and ITIL framework. Equally, this paper presents an analysis and comparison of two approaches for the implementation of the module: Drools and semantic formalism. The results of this study may be applicable to other telecommunication domains.

Convexification of bad data and topology error detection and identification problems in AC electric power systems

This study is motivated by major needs for accurate bad data detection and topology identification in the emerging electric energy systems. Due to the non-convex problem formulation, past methods usually reach a local optimum. This deficiency may lead to wrong bus/branch modelling and inappropriate noise assumption, causing significantly biased state estimate, incorrect system operation, and user cutoff. To overcome the local optimum issue, the authors propose in this study how to convexify bad data detection and topology identification problems to efficiently locate a global optimum result. To reduce relaxation error in the convexification procedure, a nuclear norm penalty is added to better approximate the original problems. Finally, they propose a new metric to evaluate the detection and identification results, which enables system operator to know how confidence one is for further system operations. Simulation results performed for several IEEE test systems show promising results for the future smart grid in improved accuracy.

Bayesian method for simultaneous gross error detection and data reconciliation

Process measurements collected from daily industrial plant operations are essential for process monitoring, control, and optimization. However, those measurements are generally corrupted by errors, which include gross errors and random errors. Conventionally, those two types of errors were addressed separately by gross error detection and data reconciliation. Solving the simultaneous gross error detection and data reconciliation problem using the hierarchical Bayesian inference technique is focused. The proposed approach solves the following problems in a unified framework. First, it detects which measurements contain gross errors. Second, the magnitudes of the gross errors are estimated. Third, the covariance matrix of the random errors is estimated. Finally, data reconciliation is performed using the maximum a posteriori estimation. The proposed algorithm is applicable to both linear and nonlinear systems. For nonlinear case, the algorithm does not involve any linearization or approximation steps. Numerical case studies are provided to demonstrate the effectiveness of the proposed method. © 2015 American Institute of Chemical Engineers AIChE J, 61: 3232–3248, 2015