Distance-based Outlier Detection Research Articles

Distance-based outlier detection for high dimension, low sample size data

ABSTRACTDespite the popularity of high dimension, low sample size data analysis, there has not been enough attention to the sample integrity issue, in particular, a possibility of outliers in the data. A new outlier detection procedure for data with much larger dimensionality than the sample size is presented. The proposed method is motivated by asymptotic properties of high-dimensional distance measures. Empirical studies suggest that high-dimensional outlier detection is more likely to suffer from a swamping effect rather than a masking effect, thus yields more false positives than false negatives. We compare the proposed approaches with existing methods using simulated data from various population settings. A real data example is presented with a consideration on the implication of found outliers.

Correcting and complementing freeway traffic accident data using Mahalanobis distance based outlier detection

A huge amount of traffic data is archived which can be used in data mining especially supervised learning. However, it is not being fully used due to lack of accurate accident information (labels). ...

A New Mining Method to Detect Real Time Substance Use Events from Wearable Biosensor Data Stream.

Detecting real time substance use is a critical step for optimizing behavioral interventions to prevent drug abuse. Traditional methods based on self-reporting or urine screening are inefficient or intrusive for drug use detection, and inappropriate for timely interventions. For example, self-report suffers from distortion or recall bias; while urine screening often detects drug use that occurred only within the previous 72 hours. Methods for real-time substance use detection are severely underdeveloped, partly due to the novelty of wearable biosensor technique and the lack of substantive clinical data for evaluation. We propose a new real-time drug use event detection method using data obtained from wearable biosensors. Specifically, this method is built upon the slide window technique to process the data stream, and a distance-based outlier detection method to identify substance use events. This novel method is designed to examine how to detect and set up the thresholds of parameters in real-time drug use event detection for wearable biosensor data streams. Our numerical analyses empirically identified the thresholds of parameters used to detect the cocaine use and showed that this proposed method could be adapted to detect other substance use events.

GPU Strategies for Distance-Based Outlier Detection

The process of discovering interesting patterns in large, possibly huge, data sets is referred to as data mining, and can be performed in several flavours, known as “data mining functions.” Among these functions, outlier detection discovers observations which deviate substantially from the rest of the data, and has many important practical applications. Outlier detection in very large data sets is however computationally very demanding and currently requires high-performance computing facilities. We propose a family of parallel and distributed algorithms for graphic processing units (GPU) derived from two distance-based outlier detection algorithms: BruteForce and SolvingSet. The algorithms differ in the way they exploit the architecture and memory hierarchy of the GPU and guarantee significant improvements with respect to the CPU versions, both in terms of scalability and exploitation of parallelism. We provide a detailed discussion of their computational properties and measure performances with an extensive experimentation, comparing the several implementations and showing significant speedups.

Distance-based outlier detection in data streams

Continuous outlier detection in data streams has important applications in fraud detection, network security, and public health. The arrival and departure of data objects in a streaming manner impose new challenges for outlier detection algorithms, especially in time and space efficiency. In the past decade, several studies have been performed to address the problem of distance-based outlier detection in data streams (DODDS), which adopts an unsupervised definition and does not have any distributional assumptions on data values. Our work is motivated by the lack of comparative evaluation among the state-of-the-art algorithms using the same datasets on the same platform. We systematically evaluate the most recent algorithms for DODDS under various stream settings and outlier rates. Our extensive results show that in most settings, the MCOD algorithm offers the superior performance among all the algorithms, including the most recent algorithm Thresh_LEAP.

Onion-Peeling Outlier Detection in 2-D data Sets

Detection is a critical and cardinal research task due its array of applications in variety of domains ranging from data mining, clustering, statistical analysis, fraud detection, network intrusion detection and diagnosis of diseases etc. Over the last few decades, distance-based outlier detection algorithms have gained significant reputation as a viable alternative to the more traditional statistical approaches due to their scalable, non-parametric and simple implementation. In this paper, we present a modified onion peeling (Convex hull) genetic algorithm to detect outliers in a Gaussian 2-D point data set. We present three different scenarios of outlier detection using a) Euclidean Distance Metric b) Standardized Euclidean Distance Metric and c) Mahalanobis Distance Metric. Finally, we analyze the performance and evaluate the results.

A Data-Mining Approach for Wind Turbine Power Generation Performance Monitoring Based on Power Curve

A new data-mining approach based on power curve profiles is put forward to monitor the power generation performance of wind turbines in this paper. Through assessing the wind-speed power datasets, the weakened power generation performance of turbines could be identified effectively by this approach. Shapes of power curve profiles over consecutive time intervals are constructed by fitting power curve models into wind-speed power datasets. In this research, we designed the Auto-adapt Optimal Interclass Variance algorithm, optimal constraint in each wind-speed power sub-dataset is explored for governing the data-driven method based on distance-based outlier detection and variance analysis model. The AOIV algorithm achieves the self-optimization of the threshold parameter and reaches a high degree of robustness to variations in wind-power generation performance monitoring. The blind industrial researches are conducted to validate the effectiveness of this approach, also indicates the decrease of error rates while detecting weakened power generation performance and the improvement of turbines’ power output.

Developing a Modified Logistic Regression Model for Diabetes Mellitus and Identifying the Important Factors of Type II Dm

Background/Objectives: Different methods can be applied to create predictive models for the clinical data with binary outcome variable. This research aims to explore the process of constructing the modified predictive model of Logistic Regression (LR). Method/Statistical Analysis: To improve the accuracy of prediction, the Distance based Outlier Detection (DBOD)is used for pre-processing and Bipolar Sigmoid Function calculated using Neuro based Weight Activation Function is used in Logistic Regression instead of Sigmoid Function. Datasets were collected from clinical laboratory of AR Hospital in Madurai for the three years 2012, 2013 and 2014 are used for analysis. Data pre-processing is done to avoid the existence of insignificant data in the dataset. The detected outliers, using DBOD method are treated using a method closest to the normal range. A comparative study among different distance measures likes Euclidean and Manhattan etc. are done for DBOD method. The pre-processed data finally is fed as input to the Logistic Regression model. Maximum likelihood estimation is used to fit the model. Logistic Model is built from the Sigmoid Function using the Regression Coefficients. The accuracy of the model is evaluated by 10 fold cross validation. Findings: Logistic Model is built from the Sigmoid Function using the Regression Coefficients, produces the accuracy of 79%. The Sigmoid Function calculated using Random Weight Function provides the prediction accuracy of 84.2% and the Bipolar Sigmoid Function calculated using Neuro based Weight Activation function provides the prediction accuracy of 90.4%. On comparison, Bipolar Sigmoid Function calculated using Neuro weight activation function outperforms well than the Sigmoid Function calculated using regression coefficients. Improvements/Applications: The accuracy of Logistic Regression is improved from 79% to 90.4%. The most important factors: Erythrocyte Sedimentation Rate (ESR) and Estimation of Mean blood Glucose are identified from positive subjects of Diabetes Mellitus. The analysis is done for the 31 Diabetes Disease attributes of three years dataset. Keywords: Bipolar Sigmoid Neuro-Weight Activation Function, Distance based Outlier Detection Method, Logistic Regression, Random Weight Function, Sigmoid Activation Function, Type 2 Diabetes Risk Factors

An Approach to Perform Uncertainity Analysis on a Spatial Dataset using Clustering and Distance based Outlier Detection Technique

Background: In past years, many methods have been implemented for maintaining and supervising uncertain data that may occur due to collection of data in new ways which results in missing values, erroneous data. The main aim of this work is to help the end user to get correct information about spatial data. Method: The behaviour of data as an outlier is the result of uncertainty. The challenge in spatial data sets is to cluster uncertain objects. Hence, unsupervised clustering can be used to deal with this type of data. In this paper, the difficulty of outlier detection with uncertain data is examined. Finding: To improve the performance and quality, Voronoi Diagram is used which partition the objects into each cell and helps to see the exact location of an object. The integral part is the pre-processing step of removing uncertainty to avoid wrong interpretation. Furthermore, CLARA (Clustering LARge Applications) algorithm is applied to produce the high quality clusters. It has an in-built function of outlier detection too and it is suitable for large data set. This algorithm uses Mahalanobis Distance to calculate the distance between cluster and its members, to remove outliers and reduce uncertainty for feasible and supporting inputs. This procedure can be a valid provision to be use in geo-database creation. Improvement: The methodology can be enhanced by designing the procedure to develop a Decision Support System (DSS) for spatial database creation.

Initialization of K-modes clustering using outlier detection techniques

The K-modes clustering has received much attention, since it works well for categorical data sets. However, the performance of K-modes clustering is especially sensitive to the selection of initial cluster centers. Therefore, choosing the proper initial cluster centers is a key step for K-modes clustering. In this paper, we consider the initialization of K-modes clustering from the view of outlier detection. We present two different initialization algorithms for K-modes clustering, where the first is based on the traditional distance-based outlier detection technique, and the second is based on the partition entropy-based outlier detection technique. By using the above two outlier detection techniques to calculate the degree of outlierness of each object, our algorithms can guarantee that the chosen initial cluster centers are not outliers. Moreover, during the process of initialization, we adopt a new distance metric — weighted matching distance metric, to calculate the distance between two objects described by categorical attributes. Experimental results on several UCI data sets demonstrate the effectiveness of our initialization algorithms for K-modes clustering.

An Efficient Algorithm for Distributed Outlier Detection in Large Multi-Dimensional Datasets

The distance-based outlier is a widely used definition of outlier. A point is distinguished as an outlier on the basis of the distances to its nearest neighbors. In this paper, to solve the problem of outlier computing in distributed environments, DBOZ, a distributed algorithm for distance-based outlier detection using Z-curve hierarchical tree (ZH-tree) is proposed. First, we propose a new index, ZH-tree, to effectively manage the data in a distributed environment. ZH-tree has two desirable advantages, including clustering property to help search the neighbors of a point, and hierarchical structure to support space pruning. We also design a bottom-up approach to build ZH-tree in parallel, whose time complexity is linear to the number of dimensions and the size of dataset. Second, DBOZ is proposed to compute outliers in distributed environments. It consists of two stages. 1) To avoid calculating the exact nearest neighbors of all the points, we design a greedy method and a new ZH-tree based k-nearest neighbor searching algorithm (ZHkNN for short) to obtain a threshold LW. 2) We propose a filter-and-refine approach, which first filters out the unpromising points using LW, and then outputs the final outliers through refining the remaining points. At last, the efficiency and the effectiveness of ZH-tree and DBOZ are testified through a series of experiments.

Reverse Nearest Neighbors in Unsupervised Distance-Based Outlier Detection

Outlier detection in high-dimensional data presents various challenges resulting from the “curse of dimensionality.” A prevailing view is that distance concentration, i.e., the tendency of distances in high-dimensional data to become indiscernible, hinders the detection of outliers by making distance-based methods label all points as almost equally good outliers. In this paper, we provide evidence supporting the opinion that such a view is too simple, by demonstrating that distance-based methods can produce more contrasting outlier scores in high-dimensional settings. Furthermore, we show that high dimensionality can have a different impact, by reexamining the notion of reverse nearest neighbors in the unsupervised outlier-detection context. Namely, it was recently observed that the distribution of points' reverse-neighbor counts becomes skewed in high dimensions, resulting in the phenomenon known as hubness. We provide insight into how some points (antihubs) appear very infrequently in k-NN lists of other points, and explain the connection between antihubs, outliers, and existing unsupervised outlier-detection methods. By evaluating the classic k-NN method, the angle-based technique designed for high-dimensional data, the density-based local outlier factor and influenced outlierness methods, and antihub-based methods on various synthetic and real-world data sets, we offer novel insight into the usefulness of reverse neighbor counts in unsupervised outlier detection.

Cluster Based Outlier Detection Algorithm for Healthcare Data

Abstract Outliers has been studied in a variety of domains including Big Data, High dimensional data, Uncertain data, Time Series data, Biological data, etc. In majority of the sample datasets available in the repository, atleast 10% of the data may be erroneous, missing or not available. In this paper, we utilize the concept of data preprocessing for outlier reduction. We propose two algorithms namely Distance-Based outlier detection and Cluster-Based outlier algorithm for detecting and removing outliers using a outlier score. By cleaning the dataset and clustering based on similarity, we can remove outliers on the key attribute subset rather than on the full dimensional attributes of dataset. Experiments were conducted using 3 built-in Health care dataset available in R package and the results show that the cluster-based outlier detection algorithm providing better accuracy than distance based outlier detection algorithm.

Lithium-Ion Battery Cell-Balancing Algorithm for Battery Management System Based on Real-Time Outlier Detection

A novel cell-balancing algorithm which was used for cell balancing of battery management system (BMS) was proposed in this paper. Cell balancing algorithm is a key technology for lithium-ion battery pack in the electric vehicle field. The distance-based outlier detection algorithm adopted two characteristic parameters (voltage and state of charge) to calculate each cell’s abnormal value and then identified the unbalanced cells. The abnormal and normal type of battery cells were acquired by online clustering strategy and bleeding circuits (R= 33 ohm) were used to balance the abnormal cells. The simulation results showed that with the proposed balancing algorithm, the usable capacity of the battery pack increased by 0.614 Ah (9.5%) compared to that without balancing.

Top-k Outlier Detection from Uncertain Data

Uncertain data are common due to the increasing usage of sensors, radio frequency identification (RFID), GPS and similar devices for data collection. The causes of uncertainty include limitations of measurements, inclusion of noise, inconsistent supply voltage and delay or loss of data in transfer. In order to manage, query or mine such data, data uncertainty needs to be considered. Hence, this paper studies the problem of top-k distance-based outlier detection from uncertain data objects. In this work, an uncertain object is modelled by a probability density function of a Gaussian distribution. The naive approach of distance-based outlier detection makes use of nested loop. This approach is very costly due to the expensive distance function between two uncertain objects. Therefore, a populated-cells list (PC-list) approach of outlier detection is proposed. Using the PC-list, the proposed top-k outlier detection algorithm needs to consider only a fraction of dataset objects and hence quickly identifies candidate objects for top-k outliers. Two approximate top-k outlier detection algorithms are presented to further increase the efficiency of the top-k outlier detection algorithm. An extensive empirical study on synthetic and real datasets is also presented to prove the accuracy, efficiency and scalability of the proposed algorithms.

Ranking with Distance based Outlier Detection Techniques: A Survey

Outlier Detection is very much popular in Data Mining field and it is an active research area due to its various applications like fraud detection, network sensor, email spam, stock market analysis, and intrusion detection and also in data cleaning. Here we will study some outlier detection technique which are mainly based on distance-based outlier detection with ranking approach and give some idea about the new technique which we will implement in future.

Monitoring the Evolution of Traffic on Main City Roads

In order to monitor the evolution of traffic on main city roads, a fixed-point data monitoring system is devised. Firstly, after qualitative traffic data (flow, speed, and occupancy) are acquired, they are then translated into the traffic qualitative state (congested or uncongested) by Fuzzy C-means Clustering algorithm. Secondly, the Congestion Evolution Index is determined using Rescaled Range Analysis of data mining. Finally, by taking a sequence pattern similarity measurement and applying condensed hierarchical clustering methods, the routine pattern is distinguished. Consequently, real-time outlier detection is realized by a distance-based outlier detection algorithm. This algorithm was successfully applied based on 11 days of fixed-point data on the eastern segment of the Shanghai North-South expressway, it is concluded that the outliers distributed in 12:10-13:20, 13:40-14:30 and 17:10-17:15 on September 30.

Efficient distance-based outlier detection on uncertain datasets of Gaussian distribution

Uncertain data management, querying and mining have become important because the majority of real world data is accompanied with uncertainty these days. Uncertainty in data is often caused by the deficiency in underlying data collecting equipments or sometimes manually introduced to preserve data privacy. This work discusses the problem of distance-based outlier detection on uncertain datasets of Gaussian distribution. The Naive approach of distance-based outlier on uncertain data is usually infeasible due to expensive distance function. Therefore a cell-based approach is proposed in this work to quickly identify the outliers. The infinite nature of Gaussian distribution prevents to devise effective pruning techniques. Therefore an approximate approach using bounded Gaussian distribution is also proposed. Approximating Gaussian distribution by bounded Gaussian distribution enables an approximate but more efficient cell-based outlier detection approach. An extensive empirical study on synthetic and real datasets show that our proposed approaches are effective, efficient and scalable.

Coding SNPs as Intrinsic Markers for Sample Tracking in Large-Scale Transcriptome Studies

Large-scale transcriptome profiling in clinical studies often involves assaying multiple samples of a patient to monitor disease progression, treatment effect, and host response in multiple tissues. Such profiling is prone to human error, which often results in mislabeled samples. Here, we present a method to detect mislabeled sample outliers using coding single nucleotide polymorphisms (cSNPs) specifically designed on the microarray and demonstrate that the mislabeled samples can be efficiently identified by either simple clustering of allele-specific expression scores or Mahalanobis distance-based outlier detection method. Based on our results, we recommend the incorporation of cSNPs into future transcriptome array designs as intrinsic markers for sample tracking.

Anomalistic sequence detection

Given a set of training instances of sequences (time series), the problem of anomalistic sequence detection is to predict whether a newly observed time series novel or normal. Anomalistic sequence detection is very useful in many monitoring applications such as video surveillance and signal recognition. In this paper, we extend existing distance-based outlier detection algorithms to address the anomaly detection problem, and propose an instance-based anomaly detection algorithm. We study the effectiveness of these algorithms on some commonly used distance measures of time series. Experiments show that the instance-based algorithm under warping distances such as DTW and EDR achieves much better accuracy than the other combination. We observe that the actual distance calculation of warping distances contributes the main bulk of computational cost of anomaly detection. To improve the efficiency, we propose a local instance summarisation approach, called VarSpace, which reduces distance calculation by summarising similar training instances. Experiments show that the VarSpace approach can improve the efficiency of instance-based anomaly detection significantly.