Existing Outlier Detection Algorithms Research Articles

Outlier detection for heterogeneous data via fuzzy [formula omitted] covering

Outlier detection is an important research topic in data mining. The existing outlier detection method does not consider ordinal data, such as heterogeneous data. Fuzzy β covering is an information granularity representation tool, which can better fit the distribution of objects in real data and characterize the differences of objects. This paper studies outlier detection for heterogeneous data via fuzzy β covering. Firstly, some new properties of fuzzy β covering are proposed. Then, the fuzzy similarity relation between any two objects in heterogeneous data is defined, and fuzzy information granules are constructed by fuzzy β covering to obtain the new fuzzy approximation accuracy. Next, the outlier factor based on fuzzy β covering is established by integrating the outlier degree and weight function of fuzzy information granules. Moreover, the corresponding algorithm (called FBCOD algorithm) is designed. Finally, the FBCOD algorithm is compared with eight existing outlier detection algorithms on 12 UCI data sets. The experimental results of AUC value, statistical test and F1 measurement show that the FBCOD algorithm has better effectiveness and flexibility than some existing algorithms.

Outlier detection using conditional information entropy and rough set theory

Outlier detection is critically important in the field of data mining. Real-world data have the impreciseness and ambiguity which can be handled by means of rough set theory. Information entropy is an effective way to measure the uncertainty in an information system. Most outlier detection methods may be called unsupervised outlier detection because they are only dealt with unlabeled data. When sufficient labeled data are available, these methods are used in a decision information system, which means that the decision attribute is discarded. Thus, these methods maybe not right for outlier detection in a a decision information system. This paper proposes supervised outlier detection using conditional information entropy and rough set theory. Firstly, conditional information entropy in a decision information system based on rough set theory is calculated, which provides a more comprehensive measure of uncertainty. Then, the relative entropy and relative cardinality are put forward. Next, the degree of outlierness and weight function are presented to find outlier factors. Finally, a conditional information entropy-based outlier detection algorithm is given. The performance of the given algorithm is evaluated and compared with the existing outlier detection algorithms such as LOF, KNN, Forest, SVM, IE, and ECOD. Twelve data sets have been taken from UCI to prove its efficiency and performance. For example, the AUC value of CIE algorithm in the Hayes data set is 0.949, and the AUC values of LOF, KNN, SVM, Forest, IE and ECOD algorithms in the Hayes data set are 0.647, 0.572, 0.680, 0.676, 0.928 and 0.667, respectively. The advantage of the proposed outlier detection method is that it fully utilizes the decision information.

Interpretable Single-dimension Outlier Detection (ISOD): An Unsupervised Outlier Detection Method Based on Quantiles and Skewness Coefficients

A crucial area of study in data mining is outlier detection, particularly in the areas of network security, credit card fraud detection, industrial flaw detection, etc. Existing outlier detection algorithms, which can be divided into supervised methods, semi-supervised methods, and unsupervised methods, suffer from missing labeled data, the curse of dimensionality, low interpretability, etc. To address these issues, in this paper, we present an unsupervised outlier detection method based on quantiles and skewness coefficients called ISOD (Interpretable Single dimension Outlier Detection). ISOD first fulfils the empirical cumulative distribution function before computing the quantile and skewness coefficients of each dimension. Finally, it outputs the outlier score. This paper’s contributions are as follows: (1) we propose an unsupervised outlier detection algorithm called ISOD, which has high interpretability and scalability; (2) massive experiments on benchmark datasets demonstrated the superior performance of the ISOD algorithm compared with state-of-the-art baselines in terms of ROC and AP.

A Novel Unsupervised Outlier Detection Algorithm Based on Mutual Information and Reduced Spectral Clustering

Outlier detection is an essential research field in data mining, especially in the areas of network security, credit card fraud detection, industrial flaw detection, etc. The existing outlier detection algorithms, which can be divided into supervised methods and unsupervised methods, suffer from the following problems: curse of dimensionality, lack of labeled data, and hyperparameter tuning. To address these issues, we present a novel unsupervised outlier detection algorithm based on mutual information and reduced spectral clustering, called MISC-OD (Mutual Information and reduced Spectral Clustering—Outlier Detection). MISC-OD first constructs a mutual information matrix between features, then, by applying reduced spectral clustering, divides the feature set into subsets, utilizing the LOF (Local Outlier Factor) for outlier detection within each subset and combining the outlier scores found within each subset. Finally, it outputs the outlier score. Our contributions are as follows: (1) we propose a novel outlier detection method called MISC-OD with high interpretability and scalability; (2) numerous experiments on 18 benchmark datasets demonstrate the superior performance of the MISC-OD algorithm compared with eight state-of-the-art baselines in terms of ROC (receiver operating characteristic) and AP (average precision).

Unsupervised Outlier Detection in IOT Using Deep VAE.

The Internet of Things (IoT) refers to a system of interconnected, internet-connected devices and sensors that allows the collection and dissemination of data. The data provided by these sensors may include outliers or exhibit anomalous behavior as a result of attack activities or device failure, for example. However, the majority of existing outlier detection algorithms rely on labeled data, which is frequently hard to obtain in the IoT domain. More crucially, the IoT’s data volume is continually increasing, necessitating the requirement for predicting and identifying the classes of future data. In this study, we propose an unsupervised technique based on a deep Variational Auto-Encoder (VAE) to detect outliers in IoT data by leveraging the characteristic of the reconstruction ability and the low-dimensional representation of the input data’s latent variables of the VAE. First, the input data are standardized. Then, we employ the VAE to find a reconstructed output representation from the low-dimensional representation of the latent variables of the input data. Finally, the reconstruction error between the original observation and the reconstructed one is used as an outlier score. Our model was trained only using normal data with no labels in an unsupervised manner and evaluated using Statlog (Landsat Satellite) dataset. The unsupervised model achieved promising and comparable results with the state-of-the-art outlier detection schemes with a precision of ≈90% and an F1 score of 79%.

Multigranulation Relative Entropy-Based Mixed Attribute Outlier Detection in Neighborhood Systems

Outlier detection is widely used in many fields, such as intrusion detection, credit card fraud detection, medical diagnosis, and so on. Existing outlier detection algorithms are mostly designed for dealing with numeric or categorical attributes. However, data usually exist in the form of mixed attributes in real-world applications. In this article, we propose a novel mixed attribute outlier detection method based on multigranulation relative entropy by employing the neighborhood rough set. First, the neighborhood system is constructed by optimizing the mixed distance metric and the radius of the statistical value. Second, the neighborhood entropy is introduced as an uncertainty measure of data. Furthermore, the three kinds of multigranulation relative entropy-based matrices are defined by three kinds of attribute sequences, and the multigranulation relative entropy-based outlier factor is integrated to indicate the outlier degree of every object. Based on the proposed outlier detection model, the corresponding algorithm is designed. Finally, the proposed algorithm is compared with other nine algorithms through experiments on public data. The experimental results show that the proposed technique is adaptive and effective.

Graph autoencoder-based unsupervised outlier detection

Outlier detection technologies play an important role in various application domains. Most existing outlier detection algorithms have difficulty detecting outliers that are mixed within normal object regions or around dense clusters. To address this problem, we propose a novel graph neural network structure called the graph autoencoder (GAE), which is capable of handling the task of outlier detection in Euclidean structured data. The GAE can perform feature value propagation in the form of a neural network that changes the distribution pattern of the original dataset, which can accurately detect outliers with low deviation. This method first converts the Euclidean structured dataset into a graph using the graph generation module, then inputs the dataset together with its corresponding graph into the GAE for training, and finally determines the top-n objects that are difficult to reconstruct in the output layer of the GAE as outliers. The results of comparing eight state-of-the-art algorithms on eight real-world datasets showed that GAE achieved the highest area under the receiver operating characteristic curve (ROC AUC) on six datasets. By comparing GAE with the autoencoder-based outlier detection algorithm, it was discovered that the proposed method improved the AUC by 16.9% on average for eight datasets.

Rough set-based entropy measure with weighted density outlier detection method

Abstract The rough set theory is a powerful numerical model used to handle the impreciseness and ambiguity of data. Many existing multigranulation rough set models were derived from the multigranulation decision-theoretic rough set framework. The multigranulation rough set theory is very desirable in many practical applications such as high-dimensional knowledge discovery, distributional information systems, and multisource data processing. So far research works were carried out only for multigranulation rough sets in extraction, selection of features, reduction of data, decision rules, and pattern extraction. The proposed approach mainly focuses on anomaly detection in qualitative data with multiple granules. The approximations of the dataset will be derived through multiequivalence relation, and then, the rough set-based entropy measure with weighted density method is applied on every object and attribute. For detecting outliers, threshold value fixation is performed based on the estimated weight. The performance of the algorithm is evaluated and compared with existing outlier detection algorithms. Datasets such as breast cancer, chess, and car evaluation have been taken from the UCI repository to prove its efficiency and performance.

An outlier detection algorithm for categorical matrix-object data

Outlier detection is a significant problem in data mining and machine learning which aims to discover objects in a data set that do not conform to well-defined notions of expected behavior. Generally, the input of the existing outlier detection algorithms is a collection of n objects and each object is described by a feature vector. However, in many real world applications, an object is not only described by one feature vector, but a number of feature vectors. In this paper, we define an object described by more than one feature vector as a matrix-object. Inspired by the concepts of cohesion and coupling in software engineering, we define the coupling of a matrix-object based on the average distance between it and other matrix-objects, and define its cohesion based on information entropy and mutual information. On this basis, the outlier factor of a matrix-object is given, and an outlier detection algorithm for categorical matrix-object data is proposed. The experimental results on real and synthetic data sets have shown that the proposed outlier detection algorithm can effectively detect outliers for the matrix-object data set compared with other algorithms.

Genetic-based Summarization for Local Outlier Detection in Data Stream

Outlier detection is one of the important tasks in data mining. Detecting outliers over streaming data has become an important task in many applications, such as network analysis, fraud detections, and environment monitoring. One of the well-known outlier detection algorithms called Local Outlier Factor (LOF). However, the original LOF has many drawbacks that can’t be used with data streams: 1- it needs a lot of processing power (CPU) and large memory to detect the outliers. 2- it deals with static data which mean that in any change in data the LOF recalculates the outliers from the beginning on the whole data. These drawbacks make big challenges for existing outlier detection algorithms in terms of their accuracies when they are implemented in the streaming environment. In this paper, we propose a new algorithm called GSILOF that focuses on detecting outliers from data streams using genetics. GSILOF solve the problem of large memory needed as it has fixed memory bound. GSILOF has two phases. First, the summarization phase that tries to summarize the past data arrived. Second, the detection phase detects the outliers from the new arriving data. The summarization phase uses a genetic algorithm to try to find the subset of points that can represent the whole original set. our experiments have been done over real datasets. Our experiments confirming the effectiveness of the proposed approach and the high quality of approximate solutions in a set of real-world streaming data.

CELOF: Effective and fast memory efficient local outlier detection in high-dimensional data streams

Outlier detection is an important and challenging problem in industrial automation, where data are often collected in large amounts but with little labeled information. To realize real-time outlier detection on data streams, many models have been proposed in the academic. However, most existing outlier detection algorithms still have two main limitations: (1) Need a large amount of memory to store data. (2) Poor detection of high-dimensional data in application scenarios. In this paper, we propose a new algorithm, called CELOF which can effectively overcome the two limitations. In CELOF, We first use information entropy to construct a new index weight calculation method, which can distinguish the influencing factors of different indexes and improve the detection accuracy of multi-dimensional data. Next, we designed a new reachable distance factor discrimination method to extract the original data information and then proposed a new strategy for outlier detection, which can greatly reduce the amount of data storage. Finally, the final experiment result shows that the CELOF algorithm has an average improvement of 15% in accuracy compared to the state-of-the-art algorithms, and the CELOF’s running time less than 1% of the original LOF. Additionally, our comprehensive experiments use different real data sets for simulation, and the results show that our algorithm can be widely used in different practical application scenarios without any prior information and data distribution.

Local projections for high-dimensional outlier detection

A novel approach for outlier detection is proposed, called local projections, which is based on concepts of the Local Outlier Factor (LOF) (Breunig et al. in Lof: identifying density-based local outliers. In: ACM sigmod record, ACM, volume 29, pp. 93–104, 2000) and ROBPCA (Hubert et al. in Technometrics 47(1):64–79, 2005). By using aspects of both methods, this algorithm is robust towards noise variables and is capable of performing outlier detection in multi-group situations. The idea is to focus on local descriptions of the observations and their neighbors using linear projections. The outlyingness of an observation is determined by a weighted distance of the observation to all identified projection spaces, with weights depending on the appropriateness of the local description. Experiments with simulated and real data demonstrate the usefulness of this method when compared to existing outlier detection algorithms.

NaNOD: A natural neighbour-based outlier detection algorithm

Outlier detection is an essential task in data mining applications which include, military surveillance, tax fraud detection, telecommunication, etc. In recent years, outlier detection received significant attention compared to other problem of discoveries. The focus on this has resulted in the growth of several outlier detection algorithms, mostly concerning the strategy based on distance or density. However, each strategy has intrinsic weaknesses. The distance-based techniques have the problem of local density, while the density-based method is recognized as having an issue of a low-density pattern. Also, most of the existing outlier detection algorithms have a parameter selection problem, which leads to poor detection results. In this article, we present an unsupervised density-based outlier detection algorithm to deal with these shortcomings. The proposed algorithm uses a Natural Neighbour (NaN) concept, to obtain a parameter called Natural Value (NV) adaptively, and a Weighted Kernel Density Estimation (WKDE) method to estimate the density at the location of an object. Besides, our proposed algorithm employed two different categories of nearest neighbours, k Nearest Neighbours (kNN), and Reverse Nearest Neighbours (RNN), which make our system flexible in modelling different data patterns. A Gaussian kernel function is adopted to achieve smoothness in the measure. Further, we use an adaptive kernel width concept to enhance the discrimination power between normal and outlier samples. The formal analysis and extensive experiments carried out on both artificial and real datasets demonstrate that this technique can achieve better outlier detection performance.

A fuzzy neighborhood rough set method for anomaly detection in large scale data

Mining Outlier in database is to find exceptional objects that deviate from the rest of the datasets. Besides classical outlier analysis algorithms, recent studies have focused on mining local outliers. The outliers that have density distribution significantly different from their neighborhood. However, the existing outlier detection algorithms suffer the drawbacks that they are inefficient in dealing with large scale datasets. In this paper, we propose a novel approach for outlier detection with voluminous data. This approach involves a neighborhood fuzzy rough set theory to rank outlier according to fuzzy membership function computed in rough approximation space. In order to improve the speed of computation, an efficient parallel computing system based on Map Reduce model is developed

Designing a Streaming Algorithm for Outlier Detection in Data Mining-An Incrementa Approach.

To design an algorithm for detecting outliers over streaming data has become an important task in many common applications, arising in areas such as fraud detections, network analysis, environment monitoring and so forth. Due to the fact that real-time data may arrive in the form of streams rather than batches, properties such as concept drift, temporal context, transiency, and uncertainty need to be considered. In addition, data processing needs to be incremental with limited memory resource, and scalable. These facts create big challenges for existing outlier detection algorithms in terms of their accuracies when they are implemented in an incremental fashion, especially in the streaming environment. To address these problems, we first propose C_KDE_WR, which uses sliding window and kernel function to process the streaming data online, and reports its results demonstrating high throughput on handling real-time streaming data, implemented in a CUDA framework on Graphics Processing Unit (GPU). We also present another algorithm, C_LOF, based on a very popular and effective outlier detection algorithm called Local Outlier Factor (LOF) which unfortunately works only on batched data. Using a novel incremental approach that compensates the drawback of high complexity in LOF, we show how to implement it in a streaming context and to obtain results in a timely manner. Like C_KDE_WR, C_LOF also employs sliding-window and statistical-summary to help making decision based on the data in the current window. It also addresses all those challenges of streaming data as addressed in C_KDE_WR. In addition, we report the comparative evaluation on the accuracy of C_KDE_WR with the state-of-the-art SOD_GPU using Precision, Recall and F-score metrics. Furthermore, a t-test is also performed to demonstrate the significance of the improvement. We further report the testing results of C_LOF on different parameter settings and drew ROC and PR curve with their area under the curve (AUC) and Average Precision (AP) values calculated respectively. Experimental results show that C_LOF can overcome the masquerading problem, which often exists in outlier detection on streaming data. We provide complexity analysis and report experiment results on the accuracy of both C_KDE_WR and C_LOF algorithms in order to evaluate their effectiveness as well as their efficiencies.

Trajectory Outlier Detection Algorithm for ship AIS Data based on Dynamic Differential Threshold

Trajectory outlier detection is one of the most important branches of data mining topics. Most existing outlier detection algorithms only utilized location of trajectory points and neglected some important factors such as speed, acceleration, and corner. In this paper, we propose a trajectory outlier detection algorithm based on dynamic differential threshold, called TODDT. Considering the influence of target motion information in a period of time, TODDT improves the traditional distance-based algorithm by combining dynamic differential threshold into outlier detection to discover more meaningful trajectory outliers. The experiments with real trajectory data sets in ship AIS (Automatic Identification System) show that TODDT algorithm performs efficiently and effectively when applied to the problem of trajectory outlier detection and the quality of trajectory data is greatly improved.

A Parameter-Free Outlier Detection Algorithm Based on Dataset Optimization Method

Recently, outlier detection has widespread applications in different areas. The task is to identify outliers in the dataset and extract potential information. The existing outlier detection algorithms mainly do not solve the problems of parameter selection and high computational cost, which leaves enough room for further improvements. To solve the above problems, our paper proposes a parameter-free outlier detection algorithm based on dataset optimization method. Firstly, we propose a dataset optimization method (DOM), which initializes the original dataset in which density is greater than a specific threshold. In this method, we propose the concepts of partition function (P) and threshold function (T). Secondly, we establish a parameter-free outlier detection method. Similarly, we propose the concept of the number of residual neighbors, as the number of residual neighbors and the size of data clusters are used as the basis of outlier detection to obtain a more accurate outlier set. Finally, extensive experiments are carried out on a variety of datasets and experimental results show that our method performs well in terms of the efficiency of outlier detection and time complexity.

KDE based outlier detection on distributed data streams in multimedia network

Multimedia networks hold the promise of facilitating large-scale, real-time data processing in complex environments. Their foreseeable applications will help protect and monitor military, environmental, safety-critical, or domestic infrastructures and resources. Cloud infrastructures promise to provide high performance and cost effective solutions to large scale data processing problems. This paper focused on the outlier detection over distributed data stream in real time, proposed kernel density estimation (KDE) based outlier detection algorithm KDEDisStrOut in Storm, firstly formalized the problem of outlier detection using the kernel density estimation technique and update the transported data incrementally between the child node and the coordinator node which reduces the communication cost. Then the paper adopted the exponential decay policy to keep pace with the transient and evolving natures of stream data and changed the weight of different data in the sliding window adaptively made the data analysis more reasonable. Theoretical analysis and experiments on Storm with synthetic and real data show that the KDEDisStrOut algorithm is efficient and effective compared with existing outlier detection algorithms, and more suitable for data streams.

RODHA: Robust Outlier Detection using Hybrid Approach

The task of outlier detection is to find the small groups of data objects that are exceptional to the inherent behavior of the rest of the data. Detection of such outliers is fundamental to a variety of database and analytic tasks such as fraud detection and customer migration. There are several approaches[10] of outlier detection employed in many study areas amongst which distance based and density based outlier detection techniques have gathered most attention of researchers. In information theory, entropy is a core concept that measures uncertainty about a stochastic event, and it means that entropy describes the distribution of an event. Because of its ability to describe the distribution of data, entropy has been applied in clustering applications in data mining. In this paper, we have developed a robust supervised outlier detection algorithm using hybrid approach (RODHA) which incorporates both the concept of distance and density along with entropy measure while determining an outlier. We have provided an empirical study of different existing outlier detection algorithms and established the effectiveness of the proposed RODHA in comparison to other outlier detection algorithms.

Robust Aggregation Algorithm in Sensor Networks

Saving energy to prolong network life is a big challenge for WSNs (wireless sensor networks) research. In-Network query can reduce the number or size of packets through processing data in intermediate nodes so as to consume energy effectively. Present aggregation algorithms suppose all the sample data are correct. The existing outlier detection algorithms regard detection rate as the primary object and do not consider energy consumption and query characteristic. So the simple combination of the two aspects can not bring good performance. By analyzing the influence of faulty and outlier readings to aggregation results, this paper puts forward a robust aggregation algorithm RAA (robust aggregation algorithm). RAA improves traditional aggregation query using reading vector to judge whether a faulty or outlier has happened. RAA deletes faulty readings, aggregates normal readings and reports outliers. Thus, customers can know the networks condition clearly. Finally, this paper compares RAA and TAGVoting which uses tiny aggregation algorithm to complete aggregation and the Voting algorithm to realize outlier detection at the same time. Experimental results show that RAA outperforms TAGVoting in terms of both energy consumption and detection rate.