Angle-based Outlier Detection Research Articles

Anomaly Detection for Power Quality Analysis Using Smart Metering Systems.

The problem of Power Quality analysis is becoming crucial to ensuring the proper functioning of complex systems and big plants. In this regard, it is essential to rapidly detect anomalies in voltage and current signals to ensure a prompt response and maximize the system's availability with the minimum maintenance cost. In this paper, anomaly detection algorithms based on machine learning, such as One Class Support Vector Machine (OCSVM), Isolation Forest (IF), and Angle-Based Outlier Detection (ABOD), are used as a first tool for rapid and effective clustering of the measured voltage and current signals directly on-line on the sensing unit. If the proposed anomaly detection algorithm detects an anomaly, further investigations using suitable classification algorithms are required. The main advantage of the proposed solution is the ability to rapidly and efficiently detect different types of anomalies with low computational complexity, allowing the implementation of the algorithm directly on the sensor node used for signal acquisition. A suitable experimental platform has been established to evaluate the advantages of the proposed method. All the different models were tested using a consistent set of hyperparameters and an output dataset generated from the principal component analysis technique. The best results achieved included models reaching 100% recall and a 92% F1 score.

An angle-based leak detection method using pressure sensors in water distribution networks

Abstract Leak detection has significant implications for the long-term stable operation of water distribution networks (WDNs). This study presented a novel leak detection method by calculating the angular variance between a pressure vector and other vectors in the database, to evaluate the presence of an anomaly in a network. The top priority for this method was to establish a reliable dataset collected from the pressure sensors, which is generated by EPANET 2.2. Numerous node water demand data in normal conditions were generated by the Monte Carlo method, and leak conditions with various leak flows were simulated by creating leak holes in the pipes. Through learning the composite normal and abnormal data in a certain proportion, the angle-based outlier detection model was employed to identify abnormal events. This angle-based method was applied in an actual WDN and the identification performance for anomalies was compared with that of previous detection methods. The results indicated that the novel method proposed in this study could significantly improve the accuracy and efficiency of leak detection compared to the threshold-based and distance-based detection methods.

Utilizing angle-based outlier detection method with sliding window mechanism to identify real-time crash risk

Developing real-time crash risk models has been a hot research topic as it could identify crash precursors and thus triggering active traffic management strategies. Currently, crash risk identification models were mainly developed based upon supervised learning techniques, which requires large sample size of historical crash data. However, crashes are rare events in the real world, where the performance of supervised learning methods can be severely degraded to deal with the imbalanced sample. Besides, the data heterogeneity issue is another critical challenge. In this study, the unsupervised learning approach has been introduced to address unbalanced samples and data heterogeneity issues, and the experimental results has verified the effectiveness of the method. Data from the Shanghai urban expressway system were utilized for the empirical analyses. Several unsupervised learning methods were tested, among which, Angle-Based Outlier Detection (ABOD) model showed the best performance with 80.4% sensitivity and 25.4% false alarm rate (FAR). Considering the varying traffic flow distribution, dynamic ABOD with sliding window is further proposed, which improves the sensitivity by 6.3% and reduces the FAR by 8.1%. Finally, the proposed model is used to construct personalized road-level models, which achieve good performance despite the small sample size and severe sample imbalance.

A Novel Framework to Detect Irrelevant Software Requirements Based on MultiPhiLDA as the Topic Model

Noise in requirements has been known to be a defect in software requirements specifications (SRS). Detecting defects at an early stage is crucial in the process of software development. Noise can be in the form of irrelevant requirements that are included within an SRS. A previous study had attempted to detect noise in SRS, in which noise was considered as an outlier. However, the resulting method only demonstrated a moderate reliability due to the overshadowing of unique actor words by unique action words in the topic–word distribution. In this study, we propose a framework to identify irrelevant requirements based on the MultiPhiLDA method. The proposed framework distinguishes the topic–word distribution of actor words and action words as two separate topic–word distributions with two multinomial probability functions. Weights are used to maintain a proportional contribution of actor and action words. We also explore the use of two outlier detection methods, namely percentile-based outlier detection (PBOD) and angle-based outlier detection (ABOD), to distinguish irrelevant requirements from relevant requirements. The experimental results show that the proposed framework was able to exhibit better performance than previous methods. Furthermore, the use of the combination of ABOD as the outlier detection method and topic coherence as the estimation approach to determine the optimal number of topics and iterations in the proposed framework outperformed the other combinations and obtained sensitivity, specificity, F1-score, and G-mean values of 0.59, 0.65, 0.62, and 0.62, respectively.

An Unsupervised Data-Driven Anomaly Detection Approach for Adverse Health Conditions in People Living With Dementia: Cohort Study.

BackgroundSensor-based remote health monitoring can be used for the timely detection of health deterioration in people living with dementia with minimal impact on their day-to-day living. Anomaly detection approaches have been widely applied in various domains, including remote health monitoring. However, current approaches are challenged by noisy, multivariate data and low generalizability.ObjectiveThis study aims to develop an online, lightweight unsupervised learning–based approach to detect anomalies representing adverse health conditions using activity changes in people living with dementia. We demonstrated its effectiveness over state-of-the-art methods on a real-world data set of 9363 days collected from 15 participant households by the UK Dementia Research Institute between August 2019 and July 2021. Our approach was applied to household movement data to detect urinary tract infections (UTIs) and hospitalizations.MethodsWe propose and evaluate a solution based on Contextual Matrix Profile (CMP), an exact, ultrafast distance-based anomaly detection algorithm. Using daily aggregated household movement data collected via passive infrared sensors, we generated CMPs for location-wise sensor counts, duration, and change in hourly movement patterns for each patient. We computed a normalized anomaly score in 2 ways: by combining univariate CMPs and by developing a multidimensional CMP. The performance of our method was evaluated relative to Angle-Based Outlier Detection, Copula-Based Outlier Detection, and Lightweight Online Detector of Anomalies. We used the multidimensional CMP to discover and present the important features associated with adverse health conditions in people living with dementia.ResultsThe multidimensional CMP yielded, on average, 84.3% recall with 32.1 alerts, or a 5.1% alert rate, offering the best balance of recall and relative precision compared with Copula-Based and Angle-Based Outlier Detection and Lightweight Online Detector of Anomalies when evaluated for UTI and hospitalization. Midnight to 6 AM bathroom activity was shown to be the most important cross-patient digital biomarker of anomalies indicative of UTI, contributing approximately 30% to the anomaly score. We also demonstrated how CMP-based anomaly scoring can be used for a cross-patient view of anomaly patterns.ConclusionsTo the best of our knowledge, this is the first real-world study to adapt the CMP to continuous anomaly detection in a health care scenario. The CMP inherits the speed, accuracy, and simplicity of the Matrix Profile, providing configurability, the ability to denoise and detect patterns, and explainability to clinical practitioners. We addressed the need for anomaly scoring in multivariate time series health care data by developing the multidimensional CMP. With high sensitivity, a low alert rate, better overall performance than state-of-the-art methods, and the ability to discover digital biomarkers of anomalies, the CMP is a clinically meaningful unsupervised anomaly detection technique extensible to multimodal data for dementia and other health care scenarios.

A comparative study on outlier detection techniques for noisy production data from unconventional shale reservoirs

Decline curve analyses (DCA) and rate transient analyses (RTA) are widely used to characterize the fluid flow through porous media and forecast future production. Oil and gas production data are routinely analyzed for history matching and optimizing the well stimulation methods in hydrocarbon exploration and production lifecycle. However, outliers add significant uncertainty and non-uniqueness to results from production data analysis. This study provides a structured and comprehensive overview of five widely used outlier detection (OD) techniques for identifying and removing outliers in production data. Each OD technique measures deviation differently and, therefore, has a different outcome even when applied to the same dataset, creating the need to test several methods and find the optimal technique for identifying and removing outliers from production data. First, we generated production data from a typical multi-fractured horizontal well using a numerical reservoir simulator and added random noise to the data. Then, we used five different OD techniques to identify the prelabeled outliers from the synthetic production data. Finally, we identified the best-performing OD algorithm by comparing the various evaluation metrics such as the mean absolute error (MAE), precision, sensitivity, and F1 score. Results showed that the angle-based OD (ABOD) had the best MAE, precision, sensitivity, and F1 score of 8%, 85%, 98%, and 0.90, respectively. The next best-performing OD technique was distance-based OD (DBOD), with MAE, precision, sensitivity, and F1 score of 16%, 71%, 100%, and 0.83, respectively. We tested the ABOD method on several field production datasets by assuming different outlier thresholds (fraction of the data points likely to be outliers). Visual inspection of the processed data showed that the ABOD method effectively identified and removed the outliers from a relatively clean dataset (outlier threshold of 20%) and a highly noisy dataset (outlier threshold of 80%). This algorithm is intuitive and can effectively identify and remove outliers from the field production data to improve production forecasting, reserves estimation, and rate transient analysis for multi-fractured horizontal oil and gas reservoirs.

Improved subspace-based and angle-based outlier detections for fuzzy datasets with a real case study

To overcome curse of dimensionality for outlier detecting in high dimensional dataset, axis-parallel subspace (SOD) and angle-based outlier detection (ABOD) methods were presented. These methods are also friendly used distance-based to detect outliers. In this regard, based on the reality of fuzzy data for explaining the world phenomena, this paper introduces an extended version of both methods for fuzzy dataset. First, the basic concepts of both methods are explained. Next we provide two metrics based on Euclidean and analytic distance to measure distance between fuzzy objects; also Cosine similarity measure formula for calculating the cosine of angle between two difference vectors in high-dimensional fuzzy dataset is illustrated. Then the algorithms to determine outliers of fuzzy datasets by using these metrics and Cosine similarity measure, based on ABOD and SOD algorithms, are presented. Some numerical experimental examples are also presented, in which both real and synthesis datasets are used, For a real numerical examination, we have applied proposed algorithms to data from 15 Iranian petrochemical companies in a fully fuzzy environment. The obtained results show the significant properties of the new methods in detecting outliers.

Performance evaluation of outlier detection techniques in production timeseries: A systematic review and meta-analysis

Time-series data have been extensively collected and analyzed in many disciplines, such as stock market, medical diagnosis, meteorology, and oil and gas industry. Numerous data in these disciplines are sequence of observations measured as functions of time, which can be further used for different applications via analytical or data analytics techniques (e.g., to forecast future price, climate change, etc.). However, presence of outliers can cause significant uncertainties to interpretation results; hence, it is essential to remove the outliers accurately and efficiently before conducting any further analysis. A total of 17 techniques that belong to statistical, regression-based, and machine learning (ML) based categories for outlier detection in timeseries are applied to the oil and gas production data analysis. 15 of these methods are utilized for production data analysis for the first time. Two state-of-the-art and high-performance techniques are then selected for data cleaning which require minimum control and time complexity. Moreover, performances of these techniques are evaluated based on several metrics including the accuracy, precision, recall, F1 score, and Cohen’s Kappa to rank the techniques. Results show that eight unsupervised algorithms outperform the rest of the methods based on the synthetic case study with known outliers. For example, accuracies of the eight shortlisted methods are in the range of 0.83–0.99 with a precision between 0.83 and 0.98, compared to 0.65–0.82 and 0.07–0.77 for the others. In addition, ML-based techniques perform better than statistical techniques. Our experimental results on real field data further indicate that the k-nearest neighbor (KNN) and Fulford-Blasingame methods are superior to other outlier detection frameworks for outlier detection in production data, followed by four others including density-based spatial clustering of applications with noise (DBSCAN), and angle-based outlier detection (ABOD). Even though the techniques are examined with oil and gas production data, but the same data cleaning workflow can be used to detect timeseries’ outliers in other disciplines.

Utilizing Angle-Based Outlier Detection Method with Sliding Window Mechanism to Identify Real-Time Crash Risk

Utilizing Angle-Based Outlier Detection Method with Sliding Window Mechanism to Identify Real-Time Crash Risk

Research on a Distribution-Outlier Detection Algorithm Based on Logistics Distribution Data

In this paper, we propose an outlier detection algorithm, which improves the existing algorithm from three aspects: the time outlier, the outlier detection of logistics distribution path, the time complexity. Aiming at the time outlier, this paper proposes an abnormal objective function based on the average speed of the logistics distribution process and realizes the detection of the outlier time of logistics distribution. In the aspect of the outlier detection of logistics distribution path, this paper combines the distance- and the angle-based outlier detection algorithms to realize the detection of the local abnormal situation of the logistics distribution trajectory. At the same time, the high time complexity of the algorithm is reduced by only retrieving the path sub-sections in the neighborhood of the proper radius. Experimental results indicate that the efficiency and speed of the proposed algorithm are better than those of the trajectory outlier detection (TRAOD) algorithm, and the proposed algorithm is suitable for outlier detection of logistics distribution.

Разработка модификации метода главных проекций Торгерсона с применением анализа кумулятивных кривых в задаче выявления выбросов в данных больших размерностей

Рассмотрена задача выявления аномальных наблюдений в данных больших размерностей на основе метода многомерного шкалирования с учетом возможности построения качественной визуализации данных. Предложен алгоритм модифицированного метода главных проекций Торгерсона, основанный на построении подпространства проектирования исходных данных путем изменения способа факторизации матрицы скалярных произведений при помощи метода анализа кумулятивных кривых. Построено и проанализировано эмпирическое распределение F -меры для разных вариантов проектирования исходных данных Purpose. Purpose of the article. The paper aims at the development of methods for multidimensional data presentation for solving classification problems based on the cumulative curves analysis. The paper considers the outlier detection problem for high-dimensional data based on the multidimensional scaling, in order to construct high-quality data visualization. An abnormal observation (or outlier), according to D. Hawkins, is an observation that is so different from others that it may be assumed as appeared in the sample in a fundamentally different way. Methods. One of the conceptual approaches that allow providing the classification of sample observations is multidimensional scaling, representing by the classical Orlochi method, the Torgerson main projections and others. The Torgerson method assumes that when converting data to construct the most convenient classification, the origin must be placed at the gravity center of the analyzed data, after which the matrix of scalar products of vectors with the origin at the gravity center is calculated, the two largest eigenvalues and corresponding eigenvectors are chosen and projection matrix is evaluated. Moreover, the method assumes the linear partitioning of regular and anomalous observations, which arises rarely. Therefore, it is logical to choose among the possible axes for designing those that allow obtaining more effective results for solving the problem of detecting outlier observations. A procedure of modified CC-ABOD (Cumulative Curves for Angle Based Outlier Detection) to estimate the visualization quality has been applied. It is based on the estimation of the variances of angles assumed by particular observation and remaining observations in multidimensional space. Further the cumulative curves analysis is implemented, which allows partitioning out groups of closely localized observations (in accordance with the chosen metric) and form classes of regular, intermediate, and anomalous observations. Results. A proposed modification of the Torgerson method is developed. The F1-measure distribution is constructed and analyzed for different design options in the source data. An analysis of the empirical distribution showed that in a number of cases the best axes are corresponding to the second, third, or even fourth largest eigenvalues. Findings. The multidimensional scaling methods for constructing visualizations of multi-dimensional data and solving problems of outlier detection have been considered. It was found out that the determination of design is an ambiguous problem.

A comparative investigation of data-driven approaches based on one-class classifiers for condition monitoring of marine machinery system

The safety and reliability of ship navigation depend heavily on the performance of the marine machinery system, which can be maintained at a high level by condition based maintenance. Intelligent condition monitoring is the key to achieve condition based maintenance. However, it is not easy to realize intelligent condition monitoring in a real ship, because there are not enough labeled fault samples to train an accurate detection model. In fact, the potential value of a large number of normal samples collected by the shipboard monitoring system has not been mined out. Since one-class classifiers only need one-class samples to train the detection models, they are very suitable for such occasions. In this paper, we investigate the performance of several representative one-class classifiers, such as OCSVM (One class support vector machine), SVDD (Support vector data description), GKNN (Global k-nearest neighbors), LOF (Local outlier factor), IForest (Isolation forest) and ABOD (Angle-based outlier detection), in condition monitoring of the marine machinery system. The dataset of a marine gas turbine propulsion system is used for a case study, which is a simulation dataset verified by the real ship data. Compared with previous literatures, our new work is to investigate the performance of a variety of key one-class classifiers, not only considering the common evaluation indexes for machine learning, but also considering the sample combinations of the training dataset, distribution of misclassified samples, and the tolerance to contaminated data. The experimental results show that these algorithms have good performance in the dataset in some common evaluation indexes. However, they show some obvious performance differences in some novel evaluation indexes we proposed. This can provide decision support for the application of one-class classifiers in the condition monitoring of many other marine machinery systems.

Big Data Cleaning Based on Mobile Edge Computing in Industrial Sensor-Cloud

With the advent of 5G, the industrial Internet of Things has developed rapidly. The industrial sensor-cloud system (SCS) has also received widespread attention. In the future, a large number of integrated sensors that simultaneously collect multifeature data will be added to industrial SCS. However, the collected big data are not trustworthy due to the harsh environment of the sensor. If the data collected at the bottom networks are directly uploaded to the cloud for processing, the query and data mining results will be inaccurate, which will seriously affect the judgment and feedback of the cloud. The traditional method of relying on sensor nodes for data cleaning is insufficient to deal with big data, whereas edge computing provides a good solution. In this article, a new data cleaning method is proposed based on the mobile edge node during data collection. An angle-based outlier detection method is applied at the edge node to obtain the training data of the cleaning model, which is then established through support vector machine. Besides, online learning is adopted for model optimization. Experimental results show that multidimensional data cleaning based on mobile edge nodes improves the efficiency of data cleaning while maintaining data reliability and integrity, and greatly reduces the bandwidth and energy consumption of the industrial SCS.

A novelty detection patent mining approach for analyzing technological opportunities

Early opportunity identification is critical for technology-based firms seeking to develop technology or product strategies for competitive advantage in the future. This research develops a patent mining approach based on the novelty detection statistical technique to identify unusual patents that may provide a fresh idea for potential opportunities. A natural language processing technique, latent semantic analysis, is applied to extract hidden relations between words in patent documents for alleviating the vocabulary mismatch problem and reducing the cumbersome efforts of keyword selection by experts. The angle-based outlier detection method, a novelty detection statistical technique, is used to determine outlier patents that are distinct from the majority of collected patent documents in a high-dimensional data space. Finally, visualization tools are developed to analyze the identified outlier patents for exploring potential technological opportunities. The developed methodology is applied in the telehealth industry and research findings can help telehealth firms formulate their technology strategies.

TagFall: Towards Unobstructive Fine-Grained Fall Detection based on UHF Passive RFID Tags

Falls are among the leading causes of hospitalization for the elderly and illness individuals. Considering that the elderly often live alone and receive only irregular visits, it is essential to develop such a system that can effectively detect a fall or abnormal activities. However, previous fall detection systems either require to wear sensors or are able to detect a fall but fail to provide fine-grained contextual information (e.g., what is the person doing before falling, falling directions). In this paper, we propose a device-free, fine-grained fall detection system based on pure passive UHF RFID tags, which not only is capable of sensing regular actions and fall events simultaneously, but also provide caregivers the contexts of fall orientations. We first augment the Angle-based Outlier Detection Method (ABOD) to classify normal actions (e.g., standing, sitting, lying and walking) and detect a fall event. Once a fall event is detected, we first segment a fix-length RSSI data stream generated by the fall and then utilize Dynamic Time Warping (DTW) based kNN to distinguish the falling direction. The experimental results demonstrate that our proposed approach can distinguish the living status before fall happening, as well as the fall orientations with a high accuracy. The experiments also show that our device-free, fine-grained fall detection system offers a good overall performance and has the potential to better support the assisted living of older people.

Research on Outlier Detection Algorithm for Evaluation of Battery System Safety

Battery system is the key part of the electric vehicle. To realize outlier detection in the running process of battery system effectively, a new high-dimensional data stream outlier detection algorithm (DSOD) based on angle distribution is proposed. First, in order to improve the algorithm stability in high-dimensional space, the method of angle distribution-based outlier detection algorithm is employed. Second, to reduce the computational complexity, a small-scale calculation set of data stream is established, which is composed of normal set and border set. For the purpose of solving the problem of concept drift, an update mechanism for the normal set and border set is developed in this paper. By this way, these hidden abnormal points will be rapidly detected. The experimental results on real data sets and battery system simulation data sets demonstrate that DSOD is more efficient than Simple variance of angles (Simple VOA) and angle-based outlier detection (ABOD) and is very suitable for the evaluation of battery system safety.

편향된 의견 문서 검출을 위한 이상치 탐지 기법

Users in social media post various types of opinions such as product reviews and movie reviews. It is a common trend that customers get assistance from the opinions in making their decisions. However, as opinion usage grows, distorted feedbacks also have increased. For example, exaggerated positive opinions are posted for promoting target products. So are negative opinions which are far from common evaluations. Finding these biased opinions becomes important to keep social media reliable. Techniques of opinion mining (or sentiment analysis) have been developed to determine sentiment polarity of opinionated documents. These techniques can be utilized for finding the biased opinions. However, the previous techniques have some drawback. They categorize the text into only positive and negative, and they also need a large amount of training data to build the classifier. In this paper, we propose methods for discovering the biased opinions which are skewed from the overall common opinions. The methods are based on angle based outlier detection and personalized PageRank, which can be applied without training data. We analyze the performance of the proposed techniques by presenting experimental results on a movie review dataset.

Detection of Profile-injection Attacks in Recommender Systems Using Outlier Analysis

Abstract E-Commerce recommender systems are vulnerable to different types of profile-injection attacks where a number of fake user profiles are inserted into the system to influence the recommendations made to the users. In this paper, we have proposed three strategies of detecting such attacks with the help of outlier analysis. In all these strategies, the attack-profiles are considered as outliers in the user rating dataset. Firstly, we have used Partition around Medoid (PAM) clustering algorithm in dete cting the attack-profiles. An incremental version of the PAM algorithm has been applied and tested for evaluating the performance of the system in identifying the attack profiles when they come into the system. Experiments show that though PAM is able to d etect attack profiles with larger number of filler items very well, a percentage of attack profiles with smaller number of filler items is not included in outlier clusters-they are included in large clusters. Secondly, we have applied a PAM-based outlier detection algorithm to find these attack profiles in large clusters. Finally, an angle based outlier detection strategy is used for fin ding attack profiles in the database under attack.