Local Outlier Detection Research Articles

Short-Term Solar Power Forecasting Based on Weighted Gaussian Process Regression

Photovoltaic (PV) power is volatile in nature and raises the level of uncertainty in power systems. PV power forecasting is an important measure to solve this problem. It helps to improve the reliability and reduces the generation cost. Advances in computer technology and sensors make the numeric modeling methods a hotspot in the field of PV power forecasting. However, data modeling methods strongly rely on the accuracy of measurement data. Unavoidable outliers in the measured meteorological data have an adverse effect on the model due to their heteroscedasticity. Although many studies can be found focusing on outlier detection, only a few have incorporated outlier detection with regression models. In this study, an innovative method employing the weighted Gaussian process regression approach is proposed, such that data samples with higher outlier potential have a low weight. A density-based local outlier detection approach is introduced to compensate the deterioration of Euclidean distance for high-dimensional data. A novel concept of the degree of nonlinear correlation is incorporated to compute the contribution of every individual data attribute. Effectiveness of the proposed method is demonstrated by performing an experimental analysis and making comparisons with other typical data-based approaches, and the results exhibit higher estimation accuracy.

Online Detection of Low-Quality Synchrophasor Measurements: A Data-Driven Approach

In this paper, an online data-driven approach is proposed for the detection of low-quality synchrophasor measurements. The proposed method leverages the spatio-temporal similarities among multiple-time-instant synchrophasor measurements and formulates the low-quality synchrophasor data as spatio-temporal outliers. A density-based local outlier detection technique is proposed to detect the spatio-temporal outliers. This data-driven approach involves no system modeling information. The detection algorithm can operate under both normal and fault-on system conditions, with fast computation speed suitable for online applications. Case studies on both synthetic and real-world synchrophasor data verify the effectiveness of the proposed approach.

LOMA: A local outlier mining algorithm based on attribute relevance analysis

Abstract In this study, we propose a novel local outlier detection approach - called LOMA - to mining local outliers in high-dimensional data sets. To improve the efficiency of outlier detection, LOMA prunes irrelevance attributes and objects in the data set by analyzing attribute relevance with a sparse factor threshold. Such a pruning technique substantially reduce the size of data sets. The core of LOMA is searching sparse subspace, which implements the particle swarm optimization method in reduced data sets. In the process of searching sparse subspace, we introduce the sparse coefficient threshold to represent sparse degrees of data objects in a subspace, where the data objects are considered as local outliers. The attribute relevance analysis provides a guidance for experts and users to identify useless attributes for detecting outliers. In addition, our sparse-subspace-based outlier algorithm is a novel technique for local-outlier detection in a wide variety of applications. Experimental results driven by both synthetic and UCI data sets validate the effectiveness and accuracy of our LOMA. In particular, LOMA achieves high mining efficiency and accuracy when the sparse factor threshold is set to a small value.

Complexity reduction in choropleth map animations by autocorrelation weighted generalization of time-series data

ABSTRACTChoropleth map animation is widely used to show the development of spatial processes over time. Although animation congruently depicts change, the rapid succession of complex map scenes easily exceeds the human cognitive capacity, causing map users to miss important information. Hence, a reduction of the visual complexity of map animations is desirable. This article builds on research related to complexity reduction of static choropleth maps. It proposes value generalization of choropleth time-series data in space and time, by using a method that adapts to the degree of global spatiotemporal autocorrelation within the dataset. A combination with upstream algorithms for local outlier detection results in less complex map animations focusing on large-scale patterns while still preserving significant local deviations in space and time. An according software application allows for in-depth exploration of the spatial and temporal autocorrelation structures in time-series data and provides control over the whole process of generalization.

A hybrid algorithm for mining local outliers in categorical data

Outlier detection is an important task in data mining. Many approaches have been developed to detect outliers. However, most researches focus on global outlier detection. In many situations, the local outlier detection is more valuable than the global outlier detection. In this paper, the existing methods for outlier detection are discussed firstly, and then the definition of local outlier and some formulas are given. Also a hybrid algorithm for mining local outlier is proposed which is based on clustering algorithm and standard deviation in statistics. By calculating the standard deviation of a cluster and local outlier factor of an object in the cluster, we can identify that the clusters with higher standard deviation may have outliers, and the objects with higher local outlier factor can be recognised as outliers. Experimental results on real datasets show that the proposed algorithm is correct and effective for mining local outliers.

An improved outlier delection algorithm K-LOF based on density

The local outlier factor (LOF) algorithm is one of the representative algorithms based on the density outlier detection algorithm. But the algorithm has the problem of high time complexity, not suitable for large data sets and high dimensional data set. Therefore, this paper proposes a new outlier detection algorithm, clustering the data sets determines the data center of data space through the K-means clustering algorithm, building data set primary model by setting the distance threshold of the data set object to the data center, and optimizing the screening process combined the neighbor distribution of data objects. Although the use of clustering algorithm for abnormal data set screening will increase the computational complexity of the algorithm, but the data center space once identified will no longer need to repeat the calculation, so with the increase of data, the advantages of the algorithm will become more and more obvious. After testing, the algorithm can effectively improve the detection accuracy of anomaly factors, and reduce the computational complexity of the algorithm, and can complete the local outlier detection.<br />

An Unsupervised Local Outlier Detection Method for Wireless Sensor Networks

Recently, wireless sensor networks (WSNs) have provided many applications, which need precise sensing data analysis, in many areas. However, sensing datasets contain outliers sometimes. Although outliers rarely occur, they seriously reduce the precision of the sensing data analysis. In the past few years, many researches focused on outlier detection. However, many of them ignored one factor that WSNs are usually deployed in a dynamic environment that changes with time. Thus, we propose a new method, which is an unsupervised learning method based on mean-shift algorithm, for outlier detection that can be used in a dynamic environment for WSNs. To make our method adapt to a dynamic environment, we define two new distances for outlier detection. Moreover, the simulation shows that our method performed on real sensing dataset has ideal results; it finds outliers with a low false positive rate and has a high recall. For generality, we also test our method on different synthetic datasets.

Fast Memory Efficient Local Outlier Detection in Data Streams

Outlier detection is an important task in data mining, with applications ranging from intrusion detection to human gait analysis. With the growing need to analyze high speed data streams, the task of outlier detection becomes even more challenging as traditional outlier detection techniques can no longer assume that all the data can be stored for processing. While the well-known Local Outlier Factor (LOF) algorithm has an incremental version, it assumes unbounded memory to keep all previous data points. In this paper, we propose a memory efficient incremental local outlier (MiLOF) detection algorithm for data streams, and a more flexible version (MiLOF_F), both have an accuracy close to Incremental LOF but within a fixed memory bound. Our experimental results show that both proposed approaches have better memory and time complexity than Incremental LOF while having comparable accuracy. In addition, we show that MiLOF_F is robust to changes in the number of data points, the number of underlying clusters and the number of dimensions in the data stream. These results show that MiLOF/MiLOF_F are well suited to application environments with limited memory (e.g., wireless sensor networks), and can be applied to high volume data streams.

Fast Memory Efficient Local Outlier Detection in Data Streams

Outlier detection is an important task in data mining, with applications ranging from intrusion detection to human gait analysis. With the growing need to analyze high speed data streams, the task ...

Comparison of local outlier detection techniques in spatial multivariate data

Outlier detection techniques in spatial data should allow to identify two types of outliers: global and local ones. Local outliers typically have non-spatial attributes that strongly differ from those observed on their neighbors. Detecting local outliers requires to be able to work locally, on neighborhoods, in order to take into account the spatial dependence between the statistical units under consideration, even though the outlyingness is usually measured on the non-spatial variables. Many procedures have been outlined in the literature, but their number reduces when one wants to deal with multivariate non-spatial attributes. In this paper, focus is on the multivariate context. A review of existing procedures is done. A new approach, based on a two-step improvement of an existing one, is also designed and compared with the benchmarked methods by means of examples and simulations.

Defective Sensor Identification for WSNs Involving Generic Local Outlier Detection Tests

The behavior of a wireless sensor network dedicated to distributed estimation tasks may be significantly altered by the presence of nodes whose sensors are defective and produce erroneous measurements. This paper proposes and analyzes the performance of two distributed algorithms to help each node in determining whether it is equipped with a defective sensor. A node first collects data from its neighborhood, processes them to decide, using some generic local outlier detection test, whether these data contain outliers and broadcasts the result. Then, it determines the status of its own sensor using its result and those received from neighboring nodes. A single-decision and an iterative algorithm for defective sensor detection are proposed. Bounds on the performance of the single-decision algorithm are derived. A theoretical analysis of the probability of error and of the equilibrium of the iterative algorithm is provided for a wide class of local outlier detection tests. The tradeoff between false alarm probability and detection probability is characterized theoretically and by simulation. MAC-layer issues, as well as the effect of packet losses are accounted for.

Spatial Outlier Detection of CO2 Monitoring Data Based on Spatial Local Outlier Factor

Spatial local outlier factor (SLOF) algorithm was adopted in this study for spatial outlier detection because of the limitations of the traditional static threshold detection. Based on the spatial characteristics of CO 2 monitoring data obtained in the carbon capture and storage (CCS) project, the K-Nearest Neighbour (KNN) graph was constructed using the latitude and longitude information of the monitoring points to identify the spatial neighbourhood of the monitoring points. Then SLOF was adopted to calculate the outlier degrees of the monitoring points and the 3σ rule was employed to identify the spatial outlier. Finally, the selection of K value was analysed and the optimal one was selected. The results show that, compared with the static threshold method, the proposed algorithm has a higher detection precision. It can overcome the shortcomings of the static threshold method and improve the accuracy and diversity of local outlier detection, which provides a reliable reference for the safety assessment and warning of CCS monitoring.

Robust Subjective Visual Property Prediction from Crowdsourced Pairwise Labels.

The problem of estimating subjective visual properties from image and video has attracted increasing interest. A subjective visual property is useful either on its own (e.g. image and video interestingness) or as an intermediate representation for visual recognition (e.g. a relative attribute). Due to its ambiguous nature, annotating the value of a subjective visual property for learning a prediction model is challenging. To make the annotation more reliable, recent studies employ crowdsourcing tools to collect pairwise comparison labels. However, using crowdsourced data also introduces outliers. Existing methods rely on majority voting to prune the annotation outliers/errors. They thus require a large amount of pairwise labels to be collected. More importantly as a local outlier detection method, majority voting is ineffective in identifying outliers that can cause global ranking inconsistencies. In this paper, we propose a more principled way to identify annotation outliers by formulating the subjective visual property prediction task as a unified robust learning to rank problem, tackling both the outlier detection and learning to rank jointly. This differs from existing methods in that (1) the proposed method integrates local pairwise comparison labels together to minimise a cost that corresponds to global inconsistency of ranking order, and (2) the outlier detection and learning to rank problems are solved jointly. This not only leads to better detection of annotation outliers but also enables learning with extremely sparse annotations.

Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection

An integrated framework for density-based cluster analysis, outlier detection, and data visualization is introduced in this article. The main module consists of an algorithm to compute hierarchical estimates of the level sets of a density, following Hartigan’s classic model of density-contour clusters and trees. Such an algorithm generalizes and improves existing density-based clustering techniques with respect to different aspects. It provides as a result a complete clustering hierarchy composed of all possible density-based clusters following the nonparametric model adopted, for an infinite range of density thresholds. The resulting hierarchy can be easily processed so as to provide multiple ways for data visualization and exploration. It can also be further postprocessed so that: (i) a normalized score of “outlierness” can be assigned to each data object, which unifies both the global and local perspectives of outliers into a single definition; and (ii) a “flat” (i.e., nonhierarchical) clustering solution composed of clusters extracted from local cuts through the cluster tree (possibly corresponding to different density thresholds) can be obtained, either in an unsupervised or in a semisupervised way. In the unsupervised scenario, the algorithm corresponding to this postprocessing module provides a global, optimal solution to the formal problem of maximizing the overall stability of the extracted clusters. If partially labeled objects or instance-level constraints are provided by the user, the algorithm can solve the problem by considering both constraints violations/satisfactions and cluster stability criteria. An asymptotic complexity analysis, both in terms of running time and memory space, is described. Experiments are reported that involve a variety of synthetic and real datasets, including comparisons with state-of-the-art, density-based clustering and (global and local) outlier detection methods.

An adaptive neighbourhood construction algorithm based on density and connectivity

A neighbourhood is a refined group of data points that are locally similar. It should be defined based on the local relations in a data set. However, selection of neighbourhood parameters is an unsolved problem for the traditional neighbourhood construction algorithms such as k-nearest neighbour and ɛ-neighbourhood. To address this issue, we introduce a novel neighbourhood construction algorithm. We assume that there is no a priori information about the data set. Different from the neighbourhood definitions in the literature, the proposed approach extracts the density, connectivity and proximity relations among the data points in an adaptive manner, i.e. considering the local characteristics of points in the data set. It is based on one of the proximity graphs, Gabriel graph. The output of the proposed approach is a unique set of neighbours for each data point. The proposed approach has the advantage of being parameter-free. The performance of the neighbourhood construction algorithm is tested on clustering and local outlier detection. The experimental results with various data sets show that, compared to the competing approaches, the proposed approach improves the average accuracy 3–66% in the neighbourhood construction, and 4–70% in the clustering. It can also detect outliers successfully.

Local Outlier Detection Algorithm Based on Coefficient of Variation

Local outliers detection is an important issue in data mining. By analyzing the limitations of the existing outlier detection algorthms, a local outlier detection algorthm based on coefficient of variation is introduced. This algorthms applies K-means which is strong in outliers searching, divides data set into sections, puts outliers and their nearing clusters into a local neighbourhood, then figures out the local deviation factor of each local neighbourhood by coefficient of variation, as a result, local outliers can more likely be found.The heoretic analysis and experimental results indicate that the method is ef fective and efficient.

An Efficient Approach for Outlier Detection with Imperfect Data Labels

The task of outlier detection is to identify data objects that are markedly different from or inconsistent with the normal set of data. Most existing solutions typically build a model using the normal data and identify outliers that do not fit the represented model very well. However, in addition to normal data, there also exist limited negative examples or outliers in many applications, and data may be corrupted such that the outlier detection data is imperfectly labeled. These make outlier detection far more difficult than the traditional ones. This paper presents a novel outlier detection approach to address data with imperfect labels and incorporate limited abnormal examples into learning. To deal with data with imperfect labels, we introduce likelihood values for each input data which denote the degree of membership of an example toward the normal and abnormal classes respectively. Our proposed approach works in two steps. In the first step, we generate a pseudo training dataset by computing likelihood values of each example based on its local behavior. We present kernel \(k\) -means clustering method and kernel LOF-based method to compute the likelihood values. In the second step, we incorporate the generated likelihood values and limited abnormal examples into SVDD-based learning framework to build a more accurate classifier for global outlier detection. By integrating local and global outlier detection, our proposed method explicitly handles data with imperfect labels and enhances the performance of outlier detection. Extensive experiments on real life datasets have demonstrated that our proposed approaches can achieve a better tradeoff between detection rate and false alarm rate as compared to state-of-the-art outlier detection approaches.

An Optimized Pruning-Based Outlier Detecting Algorithm

An Optimized Pruning-based Outlier Detecting algorithm is proposed based on the density-based outlier detecting algorithm (LOF algorithm). The calculation accuracy and the time complexity of LOF algorithm are not ideal, so two steps are taken to reduce the amount of calculation and improve the calculation accuracy for LOF algorithm. Firstly, using cluster pruning technique to preprocess data set, at the same time filtering the non-outliers based on the differences of cluster models to avoid the error pruning of outliers located at the edge of clusters, different cluster models are output by inputing multiple parameters in the DBSCAN algorithm. Secondly,optimize the query process of the neighborhood (neighbor and k-neighbor). After pruning, local outlier factors are calculated only for the data objects out of clusters. Experimental results show that the algorithm proposed in this paper can improve the outlier detection accuracy, reduce the time complexity and realize the effective local outlier detection.

Outlier detection on uncertain data based on local information

Based on local information: local density and local uncertainty level, a new outlier detection algorithm is designed in this paper to calculate uncertain local outlier factor (ULOF) for each point in an uncertain dataset. In this algorithm, all concepts, definitions and formulations for conventional local outlier detection approach (LOF) are generalized to include uncertainty information. The least squares algorithm on multi-times curve fitting is used to generate an approximate probability density function of distance between two points. An iteration algorithm is proposed to evaluate K–η–distance and a pruning strategy is adopted to reduce the size of candidate set of nearest-neighbors. The comparison between ULOF algorithm and the state-of-the-art approaches has been made. Results of several experiments on synthetic and real data sets demonstrate the effectiveness of the proposed approach.

Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection

Outlier detection research has been seeing many new algorithms every year that often appear to be only slightly different from existing methods along with some experiments that show them to "clearly outperform" the others. However, few approaches come along with a clear analysis of existing methods and a solid theoretical differentiation. Here, we provide a formalized method of analysis to allow for a theoretical comparison and generalization of many existing methods. Our unified view improves understanding of the shared properties and of the differences of outlier detection models. By abstracting the notion of locality from the classic distance-based notion, our framework facilitates the construction of abstract methods for many special data types that are usually handled with specialized algorithms. In particular, spatial neighborhood can be seen as a special case of locality. Here we therefore compare and generalize approaches to spatial outlier detection in a detailed manner. We also discuss temporal data like video streams, or graph data such as community networks. Since we reproduce results of specialized approaches with our general framework, and even improve upon them, our framework provides reasonable baselines to evaluate the true merits of specialized approaches. At the same time, seeing spatial outlier detection as a special case of local outlier detection, opens up new potentials for analysis and advancement of methods.