Outlier Detection Model Research Articles

Minimum-variance-based outlier detection method using forward-search model error in geodetic networks

Abstract. Geodetic observations are crucial for monitoring landslides, crustal movements, and volcanic activity. They are often integrated with data from interdisciplinary studies, including paleo-seismological, geological, and interferometric synthetic aperture radar observations, to analyze earthquake hazards. However, outliers in geodetic observations can significantly impact the accuracy of estimation results if not reliably identified. Therefore, assessing the outlier detection model's reliability is imperative to ensure accurate interpretations. Conventional and robust methods are based on the additive bias model, which may cause type-I and type-II errors. However, outliers can be regarded as additional unknown parameters in the Gauss–Markov model. It is based on modeling the outliers as unknown parameters, considering as many combinations as possible of outliers selected from the observation set. In addition, this method is expected to be more effective than conventional methods as it is based on the principle of minimal variance and eliminates the interdependence of decisions made in iterations. The primary purpose of this study is to seek an efficient outlier detection model in the geodetic networks. The efficiency of the proposed model was measured and compared with the robust and conventional methods by the mean success rate (MSR) indicator of different types and magnitudes of outliers. Thereby, this model enhances the MSR by almost 40 %–45 % compared to the Baarda and Danish (with the variance unknown case) method for multiple outliers. Besides, the proposed model is 20 %–30 % more successful than the others in the low-controllability observations of the leveling network.

Suppression for Phase Error of Fringe Projection Profilometry Using Outlier-Detection Model: Development of an Easy and Accurate Method for Measurement

Fringe projection is an important technology in three-dimensional measurement and target recognition. The measurement accuracy depends heavily on the calibration of the absolute phase and projector pixels. An easy-to-implement calibration method based on the Random Sample Consensus (RANSAC) algorithm is proposed to exterminate the phase error data and elevate the measurement accuracy in a fringe projection system. The reconstruction experiments of a double-sphere standard demonstrate that the uncertainties in radius and sphere-distance measurement are reduced to one thousandth of the measured value or even less, and the standard deviation in multiple measurements is restricted to within 50 μm. The measurement accuracy provided by the proposed RANSAC method can be improved by up to 44% compared with that provided by traditional least squared method (LSM). The proposed calibration method is easy and simple to implement, and it does not need additional hardware, but rather a calibration board.

Unsupervised machine learning model for predicting anomalies in subsurface safety valves and application in offshore wells during oil production

Predicting oil well behavior regarding the integrity of its equipment during production and anticipating behavioral changes and anomalies are among the main challenges in oil production. In this context, this study focuses on the development of predictive models for real-time monitoring of well behavior using sensor data from production wells. An unsupervised Novelty and Outlier Detection model has been introduced with a specific focus on predicting instances of unexpected subsurface safety valve closures in subsea wells. This model effectively classifies anomalies observed in these systems by leveraging real-world pressure and temperature data sourced from published literature. The methodology involves the implementation of a floating window for assembling training and test sets. Additionally, a comprehensive investigation is conducted into the impact of hyperparameters and the model’s threshold value (cp threshold). The results highlight the effectiveness of the developed model, observed through the accuracy achieved around 99.9% in predicting spurious closure events of the Downhole Safety Valve. On the same dataset, previous works reported 99.9% accuracy by using long short-term memory (LSTM) autoencoder, 87.1% by using random forest, and 60% with the Decision Tree method. Looking at F1-SCORE values, the developed model performs the best, followed by the LSTM model, both of which are significantly superior to the Decision Tree and random forest models. Furthermore, the model’s applicability is validated through testing in ultradeep water subsea wells within the pre-salt area of the Santos Basin. The significance lies in the potential for this research to enhance anomaly prediction in offshore wells, consequently reducing the costly interventions due to equipment malfunctions. Timely detection and corrective actions, facilitated by the model, can mitigate production loss and safeguard well integrity, addressing critical concerns in the oil and gas industry.

An angle-based leak detection method using pressure sensors in water distribution networks

Abstract Leak detection has significant implications for the long-term stable operation of water distribution networks (WDNs). This study presented a novel leak detection method by calculating the angular variance between a pressure vector and other vectors in the database, to evaluate the presence of an anomaly in a network. The top priority for this method was to establish a reliable dataset collected from the pressure sensors, which is generated by EPANET 2.2. Numerous node water demand data in normal conditions were generated by the Monte Carlo method, and leak conditions with various leak flows were simulated by creating leak holes in the pipes. Through learning the composite normal and abnormal data in a certain proportion, the angle-based outlier detection model was employed to identify abnormal events. This angle-based method was applied in an actual WDN and the identification performance for anomalies was compared with that of previous detection methods. The results indicated that the novel method proposed in this study could significantly improve the accuracy and efficiency of leak detection compared to the threshold-based and distance-based detection methods.

Detecting outliers beyond tolerance limits derived from statistical process control in patient-specific quality assurance.

Tolerance limit is defined on pre-treatment patient specific quality assurance results to identify "out of the norm" dose discrepancy in plan. An out-of-tolerance plan during measurement can often cause treatment delays especially if replanning is required. In this study, we aim to develop an outlier detection model to identify out-of-tolerance plan early during treatment planning phase to mitigate the above-mentioned risks. Patient-specific quality assurance results with portal dosimetry for stereotactic body radiotherapy measured between January 2020 and December 2021 were used in this study. Data were divided into thorax and pelvis sites and gamma passing rates were recorded using 2%/2mm, 2%/1mm, and 1%/1mm gamma criteria. Statistical process control method was used to determine six different site and criterion-specific tolerance and action limits. Using only the inliers identified with our determined tolerance limits, we trained three different outlier detection models using the plan complexity metrics extracted from each treatment field-robust covariance, isolation forest, and one class support vector machine. The hyperparameters were optimized using the F1-score calculated from both the inliers and validation outliers' data. 308 pelvis and 200 thorax fields were used in this study. The tolerance (action) limits for 2%/2mm, 2%/1mm, and 1%/1mm gamma criteria in the pelvis site are 99.1% (98.1%), 95.8% (91.1%), and 91.7% (86.1%), respectively. The tolerance (action) limits in the thorax site are 99.0% (98.7%), 97.0% (96.2%), and 91.5% (87.2%). One class support vector machine performs the best among all the algorithms. The best performing model in the thorax (pelvis) site achieves a precision of 0.56 (0.54), recall of 1.0 (1.0), and F1-score of 0.72 (0.70) when using the 2%/2mm (2%/1mm) criterion. The model will help the planner to identify an out-of-tolerance plan early so that they can refine the plan further during the planning stage without risking late discovery during measurement.

A systematic review on big data applications and scope for industrial processing and healthcare sectors

Nowadays, big data is an emerging area of computer science. Data are generated through different sources such as social media, e-commerce, blogs, banking, healthcare, transactions, apps, websites, opinion platforms, etc. It is processed for effective utilization in different industries, including healthcare. These enormous generated data are essential for data analysis and processing for industrial needs. This paper reviews the work of various authors who have contributed to data collection, analyzing, processing, and viewing to explore the importance and possibilities of big data in industrial processing applications and healthcare sectors. It identifies different opportunities and challenges (data cleaning, missing values, and outlier analysis) along with applications and features of big data. This systematic review further proposed dirty data detection and cleaning and outlier detection models that can be used for many applications. The data cleaning and outlier detection models use the optimizations concept to solve the optimal centroid selection problem and suspected data.

P-775 Responsible Artificial Intelligence in Clinical Decision Making: Anomaly Detection in Trigger Day Optimization

Abstract Study question Is there a benefit of using outlier detection when developing an AI algorithm for predicting the number of oocytes retrieved in different trigger days? Summary answer Using an outlier detection as a pre-step for other algorithms helps to determine if a prediction\decision should be made for a specific scenario. What is known already Outlier detection is a process that helps in identifying unusual or unexpected observations in the data. Outliers can arise for a variety of reasons, such as: measurement errors, data entry mistakes, genuine rare events or data the model did not observe when training. In recent years multiple AI algorithms have been developed across a variety of areas in reproductive health (embryos, sperm, etc.), however not many publications have been made on the importance of implementing outlier detection when assisting doctors in clinical decisions. Study design, size, duration The data used for developing this algorithm consists of 9,618 antagonist protocol cycles performed in a large center serving over 50 physicians, between August 2017 and November 2022. The data was divided into three subsets, representing cases in which the physician performed the trigger 0, 1 or 2 days after a blood and ultrasound test day. Participants/materials, setting, methods Three outlier detection models were developed for a pre-developed trigger day selection algorithm. Each outlier model was developed for a different trigger day subset. Those models were built using Local Outlier Factor algorithm, an unsupervised method. Each model was applied to its specific data set that had been standardized and preprocessed using principal component analysis (PCA). Additionally, A rule-based outlier detection method was implemented, based on thresholds for different hormones. Main results and the role of chance The three outlier detection models were evaluated using train, validation, and test sets, identifying approximately 1.5%, 1.5%, and 2% of the data as outliers, respectively. Additionally, each model was evaluated using other datasets, for example, the "trigger today" outlier detection model was evaluated using "trigger tomorrow" and "trigger in two days" datasets, which resulted in ∼12% and ∼87% of the data being identified as outliers. To further assess the models, additional subsets were created for different trigger times, such as in 3 days, 4 days, 5 days, and 6 or more days. The results of the "trigger today" outlier detection model were highly accurate in identifying outliers, with at least 99.5% of the data being detected as outliers in all of the additional subsets. Instances with a closer relative trigger time will result in similar instances, therefore, detecting a lower percentage of outliers. Limitations, reasons for caution The outlier detection models were developed specifically for antagonist protocol cycles. Additionally, there may be rare cycles that occur infrequently and may not be well represented in the data, resulting in them being incorrectly identified as outliers, even though they are a legitimate scenario. Wider implications of the findings Outlier detection is a crucial aspect of AI algorithms to ensure accurate results. It recognizes cases that are outside of the model's training set as well as abnormal values and typos, thus avoiding predictions on untested cases. This is especially important when first deploying algorithms on diverse populations. Trial registration number Not applicable

Three-way decision-based co-detection for outliers

Outlier detection is an important research topic in data mining and machine learning. However, existing unsupervised outlier detection methods suffer from irrelevant and redundant attributes in high-dimensional data, and their performance is also limited by their outlier detection models that rely on only one view. In this study, we propose a three-way decision-based co-detection model for unsupervised outlier detection. Specifically, we first improve the local outlier factor (LOF) method by introducing the Gaussian kernel function to make the measure of local reachability density more accurate. Then, we introduce fuzzy rough sets to perform attribute reduction, which further reduces the negative effect of irrelevant and redundant attributes on the measure of sample similarity. Finally, we develop a co-detection model that is trained on the original view and the transformed view generated by principal component analysis and uses the strategy of the three-way decision to collaboratively detect outliers. The results of comparative experiments on the selected UCI datasets show that the proposed model outperforms state-of-the-art methods in terms of AUC-ROC index.

Unsupervised outlier detection for time-series data of indoor air quality using LSTM autoencoder with ensemble method

The proposed framework consists of three modules as an outlier detection method for indoor air quality data. We first use a long short-term memory autoencoder (LSTM-AE) based reconstruction error detector, which designs the LSTM layer in the shape of an autoencoder, to build a reconstruction error-based outlier detection model and extract latent features. The latent feature class-assisted vector machine detector constructs an additional outlier detection model using previously extracted latent features. Finally, the ensemble detector combines the two independent classifiers to define a new ensemble-based decision rule. Furthermore, because real-time anomaly detection proceeds with unsupervised learning, more stable and consistent external detection rules are defined than when using a single ensemble model. Laboratory tests with five random cases were performed for objective evaluation. Thus, we propose a framework that can be applied to various industrial environments by detecting and defining stable outlier decision rules.

Clustering ensemble-based novelty score for outlier detection

Recently, One-class classification algorithms have been successfully used for outlier detection problems in several industrial fields. However, in case of that the target class has complex structures, single outlier detection model with one-class classifier often poorly performs because it cannot appropriately reflect intrinsic data structures. To address this limitation, we propose a clustering ensemble-based novelty score algorithm. The proposed algorithm calculates novelty score from the mixture of multiple clustering solutions generated by both random subspace and random-K ensemble approaches. Then, final ensemble novelty score is defined by summarizing multiple novelty scores obtained from individual clustering results. Because these multiple novelty scores are computed from many possible characteristics of target class information, the proposed ensemble novelty score can appropriately reflect the inherent structures of target class. Experiments were conducted on various benchmark datasets to compared with existing methods and investigate the properties of the proposed algorithm. The experimental results confirm that the proposed algorithm outperforms existing one-class classification methods in various cases.

A novel outlier detection approach based on formal concept analysis

Outlier detection is a major research field for data mining. In recent years, rough set and granular computing have been successfully applied to outlier detection, and a series of excellent results have been achieved. However, as another important area of granular computing, the study of formal concept analysis applied to outlier detection has not been reported. We introduce formal concept analysis to deal with the outlier detection problem and propose a granular concept-based outlier detection model in this paper. Firstly, we define a distance measure between objects using the object belonging or not belonging to the extent of a granular concept. Secondly, a distance measure between granular concepts and an outlier degree of each granular concept are defined. And then an outlier degree of each object is constructed from the outlier degree of the granular concept by weighted average. Finally, a granular concept-based outlier detection model is given and the related algorithm is designed. Experiments are performed on fifteen public datasets, and then our algorithm is compared with classical and rough set-based outlier detection algorithms. The results of the experiments show that the proposed algorithm can effectively and efficiently detect outliers.

Low cost, LoRa based river water level data acquisition system

In recent years, climate change and catchment degradation have negatively affected stage patterns in rivers which in turn have affected the availability of enough water for various ecosystems. To realize and quantify the effects of climate change and catchment degradation on rivers, water level monitoring is essential. Various effective infrastructures for river water level monitoring that have been developed and deployed in developing countries over the years, are often bulky, complex and expensive to build and maintain. Additionally, most are not equipped with communication hardware components which can enable wireless data transmission. This paper presents a river water level data acquisition system that improves on the effectiveness, size, deployment design and data transmission capabilities of systems being utilized. The main component of the system is a river water level sensor node. The node is based on the MultiTech mDot – an ARM-Mbed programmable, low power RF module – interfaced with an ultrasonic sensor for data acquisition. The data is transmitted via LoRaWAN and stored on servers. The quality of the stored raw data is controlled using various outlier detection and prediction machine learning models. Simplified firmware and easy to connect hardware make the sensor node design easy to develop. The developed sensor nodes were deployed along River Muringato in Nyeri, Kenya for a period of 18 months for continuous data collection. The results obtained showed that the developed system can practically and accurately obtain data that can be useful for analysis of river catchment areas.

Utilizing angle-based outlier detection method with sliding window mechanism to identify real-time crash risk

Developing real-time crash risk models has been a hot research topic as it could identify crash precursors and thus triggering active traffic management strategies. Currently, crash risk identification models were mainly developed based upon supervised learning techniques, which requires large sample size of historical crash data. However, crashes are rare events in the real world, where the performance of supervised learning methods can be severely degraded to deal with the imbalanced sample. Besides, the data heterogeneity issue is another critical challenge. In this study, the unsupervised learning approach has been introduced to address unbalanced samples and data heterogeneity issues, and the experimental results has verified the effectiveness of the method. Data from the Shanghai urban expressway system were utilized for the empirical analyses. Several unsupervised learning methods were tested, among which, Angle-Based Outlier Detection (ABOD) model showed the best performance with 80.4% sensitivity and 25.4% false alarm rate (FAR). Considering the varying traffic flow distribution, dynamic ABOD with sliding window is further proposed, which improves the sensitivity by 6.3% and reduces the FAR by 8.1%. Finally, the proposed model is used to construct personalized road-level models, which achieve good performance despite the small sample size and severe sample imbalance.

Bayes-based air pollution computer measurement and big data outlier detection model

In today’s rapidly changing social environment, high-dimensional data abound, but not all data are meaningful. In order to reduce costs and reduce consumption, people are more interested in some important variables. Therefore, variable selection of high-dimensional data is an important research direction. Based on the Bayesian method, it stands out among many variable selection methods with effective estimation efficiency and flexible mechanism, and it is also suitable for parametric and non-parametric models. Thanks to the flexibility of the statistical conclusion inference process, combined with prior information, the actual results are more accurate. At present, many indoor pollution problems, including decoration pollution, are getting more and more attention, and its air pollution calculations are obviously very important. Indoor air quality not only directly affects the comfort of the human body, but also significantly affects the work efficiency of family members. As a result of the development of the new digital economy, digital mining technology based on big data has brought unprecedented new development methods to various fields. This paper establishes an outlier detection model, identifies data processes that are significantly different from the normal data set or expected value, and cleans the data to improve the accuracy of the integrated detection model. This paper proposes a solution to this problem by combining big data mining technology with the establishment of an outlier detection model. At the same time, it optimizes the design of the model to reduce errors, so as to realize its accurate application in the measurement of air pollutants, and aims to promote its development.

The quality inspection method of piston compressor assisted with the XGBOD model

For most manufacturers of piston compressors, all of the compressors should be tested and some key parameters are measured for the procedure of quality inspection. Traditionally, quality inspection of compressors is based on the thresholds of the key parameters (i.e., based on rules). However, due to the operation conditions of the production line varying slightly and uncertainty of measurement, the strictly fixed thresholds can often lead to misjudgment of quality inspection. To improve the correctness of quality inspection, this paper proposed a quality inspection method assisted with the Extreme Gradient Boosting Outlier Detection (XGBOD) model. XGBOD model is set up by learning anomalous patterns from historical test data, and it is used to identify finally whether it is qualified for the compressor that has been judged as unqualified by rules. The verification results demonstrate the effectiveness of the proposed quality inspection method. The XGBOD algorithm has a lower misjudgment rate than other anomaly detection algorithms. Assisted with the XGBOD model, the misjudgment rates of the proposed quality inspection method for Type-A and B compressors are reduced significantly from 15.75 % and 18.57 % to 1.18 % and 0.72 % respectively. Therefore, the quality inspection method assisted with the XGBOD model can effectively improve the correctness of the quality inspection.

A benchmarking approach for characterizing providers’ patterns of treating patients with substance use disorder

It is difficult to identify fraud, waste, and abuse (FWA) in substance use disorder (SUD) treatment. Insurers use outlier detection models to detect SUD-related FWA but these models require credible benchmarks based on cross-payer claims data. To our knowledge, these benchmarks have not been established for SUD-related services frequently cited in fraud cases — drug screening, psychotherapy, and partial hospitalization. We used de-identified and aggregated data from FAIR Health’s national database of commercial claims, including ACA marketplace plans, from 2014 to 2018 to establish regional benchmarks. We provide additional benchmarks for providers in specialties that most frequently submit claims for patients with SUD. These benchmarks can either be used to determine if a provider’s frequency of claims per-patient-per-month is unusual, potentially signaling FWA, or within a common range for the region or specialty. Our findings can be used to improve detection of FWA – a critical step towards improving the quality of treatment.

An Integrated Multilayered Framework for IoT Security Intrusion Decisions

Security breaches can seriously harm the Internet of Things (IoT) and Industrial IoT (IIoT) environments. The damage can exceed financial and material losses to threaten human lives. Overcoming these security risks is challenging given IoT ubiquity, complexity, and restricted resources. Security intrusion management is a cornerstone in fortifying the defensive security process. This paper presents an integrated multilayered framework facilitating the orchestration of the security intrusion management process and developing security decision support systems. The proposed framework incorporates four layers with four dedicated processing phases. This paper focuses mainly on the analytical layer. We present the architecture and models for predictive intrusion analytics for reactive and proactive defense strategies. We differentiate between the device and network levels to master the complexity of IoT infrastructure. Benefiting from the singularity of IIoT devices traffic, we approach the reactive security intrusion prediction through outlier detection models mean. We thoroughly experiment with ten outlier detection models on the IIoT wustl realistic dataset. The obtained results show the adequacy of the approach with an area under the curve (AUC) results surpassing 98% for several models with a good level of precision and time efficiency. Furthermore, we investigate the use of survival analysis semi-parametric predictive models to forecast the security intrusion before its occurrence for the proactive security strategy. The experiments show encouraging results with a concordance index (c-Index) reaching 89% and an integrated brier score (IBS) of 0.02. By integrating outlier intrusion detection and survival forecasting, the framework provides a valuable means to monitor the security intrusions in IoT.

A method for detecting abnormal behavior of ships based on multi-dimensional density distance and an abnormal isolation mechanism.

Abnormal ship behavior detection is essential for maritime navigation safety. Most existing abnormal ship behavior detection methods only build A ship trajectory position outlier detection model; however, the construction of a ship speed outlier detection model is also significant for maritime navigation safety. In addition, in most existing methods for detecting a ship's abnormal behavior based on abnormal thresholds, one unsuitable threshold leads to the risk of the ship not being minimized as much as possible. In this paper, we proposed an abnormal ship behavior detection method based on distance measurement and an isolation mechanism. First, to address the problem of traditional trajectory compression methods and density clustering methods only using ship position information, the minimum description length principle based on acceleration (AMDL) algorithm and Multi-Dimensional Density Clustering (MDDBSCAN) algorithm is used in this study. These algorithms not only considered the position information of the ship, but also the speed information. Second, regarding the issue of the difficulty in determining the anomaly threshold, one method for determining the anomaly threshold based on the relationship between the velocity weights and noise points of the MDDBSCAN algorithm has been introduced. Finally, due to the randomness issue of the selected segmentation value in iForest, a strategy of selectively constructing isolated trees was proposed, thus further improving the efficiency of abnormal ship behavior detection. The experimental results on the historical automatic identification system data set of Xiamen port prove the practicality and effectiveness of our proposed method. Our experiment results show that the proposed method achieves an improvement of about 10% over the trajectory outlier detection based on the local outlier fraction method, about 14% over the isolation-based online anomalous trajectory method in terms of the accuracy of ship position information anomaly detection, and about 3% over the feature fusion method in terms of the accuracy of ship speed anomaly detection. This method improves algorithm efficiency by about 5% compared to the traditional isolation forest anomaly detection algorithm.

A trajectory outlier detection method based on variational auto-encoder.

Trajectory outlier detection can identify abnormal phenomena from a large number of trajectory data, which is helpful to discover or predict potential traffic risks. In this work, we proposed a trajectory outlier detection model based on variational auto-encoder. First, the model encodes the trajectory data as parameters of distribution functions based on the statistical characteristics of urban traffic. Then, an auto-encoder network is built and trained. The training goal of the auto-encoder network is to maximize the generation probability of original trajectories when decoding. Once the model training is completed, we can detect the trajectory outlier by the difference between a trajectory and the trajectory generated by the model. The advantage of the proposed model is that it only needs to compute the difference between the original trajectory and the trajectory generated by the model when detecting the trajectory outlier, which greatly reduces the amount of calculation and makes the model very suitable for real-time detection scenarios. In addition, the distance threshold between the abnormal trajectory and the normal trajectory can be set by referring to the proportion of the abnormal trajectory in the training data set, which eliminates the difficulty of setting the threshold manually and makes the model more convenient to be applied in different actual scenes. In terms of effect, the proposed model has achieved more than 95% in accuracy, which is better than the two typical density-based and classification-based detection methods, and also better than the methods based on machine learning in recent years. In terms of efficiency, the model has good convergence in the training phase and the training time increases slowly with the data scale, which is better than or as the same as the comparison methods.

Retracted: Application of Outlier Detection Model in Korean Language and Culture Communication System Based on Artificial Intelligence.

[This retracts the article DOI: 10.1155/2022/6414112.].