Distance-based Outlier Detection Research Articles

AT-densenet with salp swarm optimization for outlier prediction

In the field of data mining, outlier prediction detects objects that behave differently from normal objects. The conventional density and distance based outlier detection cannot identify the outliers present generally in the stream data. Transfer learning applies the knowledge of Artificial Intelligence to the task of outlier prediction. It makes training faster than earlier stages of the deep learning model. However, the correlations between spatial points are not detected with the Transfer learning based approach. It can be resolved with attention based approach, which focuses on the most informative data points. Hence, this work introduces Transfer learning with an attention mechanism based deep learning model for outlier detection. The attention transfer-DenseNet model is utilized for the outlier prediction. The attention mechanism highlights outlier information in feature maps and enhances the prediction performance. Further, the attention transfer-DenseNet weights are optimized by the metaheuristic algorithm salp swarm optimization algorithm. The performance of the proposed outlier prediction is compared over the conventional models and attained better mean average precision of 0.98 (Glass dataset) and 0.986 (Abalone dataset), respectively.

Human Activity Recognition with Smartwatch Data by using Mahalanobis Distance-Based Outlier Detection and Ensemble Learning Methods

Recognition of human activities is part of smart healthcare applications. In this context, the detection of human actions with high accuracy has been a field that has been working for many years. With the increase in the usage of smart devices, smartphones and smartwatches have become the constant equipment of these studies thanks to their internal sensors. Sometimes abnormal data are included in data sets due to the way the data were collected and for reasons arising from the sensors. For this reason, it becomes important to detect outlier data. In this study, step counter and heart rate sensors were used in addition to an accelerometer and gyroscope in order to detect human activities. Afterward, the outliers were detected and cleared with a Mahalanobis distance-based approach. With the aim of achieving a better classification performance, machine learning methods were used by strengthening them with ensemble learning methods. The obtained results showed that step counter, heart rate sensors, and ensemble learning methods positively affect the success of the classification. In addition, it was found that the Mahalanobis distance-based outlier detection method increased the classification accuracy significantly.

A Probabilistic Transformation of Distance-Based Outliers

The scores of distance-based outlier detection methods are difficult to interpret, and it is challenging to determine a suitable cut-off threshold between normal and outlier data points without additional context. We describe a generic transformation of distance-based outlier scores into interpretable, probabilistic estimates. The transformation is ranking-stable and increases the contrast between normal and outlier data points. Determining distance relationships between data points is necessary to identify the nearest-neighbor relationships in the data, yet most of the computed distances are typically discarded. We show that the distances to other data points can be used to model distance probability distributions and, subsequently, use the distributions to turn distance-based outlier scores into outlier probabilities. Over a variety of tabular and image benchmark datasets, we show that the probabilistic transformation does not impact outlier ranking (ROC AUC) or detection performance (AP, F1), and increases the contrast between normal and outlier score distributions (statistical distance). The experimental findings indicate that it is possible to transform distance-based outlier scores into interpretable probabilities with increased contrast between normal and outlier samples. Our work generalizes to a wide range of distance-based outlier detection methods, and, because existing distance computations are used, it adds no significant computational overhead.

Rapid Detection of Soil Texture Attribute based on Mid-Infrared Spectral Library In Salt Affected Soils of Hungary

Quantifying among others the soil's physical properties is essential for the assessment of the diverse soil environmental functions including water balance of soils and pore structure, water erosion and various soil hydraulic properties. The mid-infrared (MIR) spectroscopy is a useful technique to predict soil attributes with high accuracy, efficiency and low cost. In this study, we examined the ability of our MIR soil spectral library in predicting the clay, silt, sand content of salt affected Hungarian soils. This research is part of a project to establish a MIR spectral library in the frame of the Hungarian Soil Information and Mentoring System (SIMS) survey. Salt affected soils type data was extracted from the spectral library then transformation of spectral reflectance values to absorbance values were performed. Moving average filtering method was applied to absorbance spectra before performing principal components analysis. To determine outlier samples and to select the proper samples for model calibration, Mahalanobis distance-based outlier detection method and Kennard-Stone Sampling selection method were applied on the principal component scores. Spectral and reference soil data were combined and split into training and testing datasets. MIR prediction models were built for sand, clay, and silt content using Partial Least Square Regression (PLSR) method. Coefficient determination, root mean square error and ratio performance to deviation were used to assess the models performance. The prediction accuracies of calibration sets for soil physical texture were excellent while the validation results were slightly lower but still with a good level of prediction.

A comparative study on outlier detection techniques for noisy production data from unconventional shale reservoirs

Decline curve analyses (DCA) and rate transient analyses (RTA) are widely used to characterize the fluid flow through porous media and forecast future production. Oil and gas production data are routinely analyzed for history matching and optimizing the well stimulation methods in hydrocarbon exploration and production lifecycle. However, outliers add significant uncertainty and non-uniqueness to results from production data analysis. This study provides a structured and comprehensive overview of five widely used outlier detection (OD) techniques for identifying and removing outliers in production data. Each OD technique measures deviation differently and, therefore, has a different outcome even when applied to the same dataset, creating the need to test several methods and find the optimal technique for identifying and removing outliers from production data. First, we generated production data from a typical multi-fractured horizontal well using a numerical reservoir simulator and added random noise to the data. Then, we used five different OD techniques to identify the prelabeled outliers from the synthetic production data. Finally, we identified the best-performing OD algorithm by comparing the various evaluation metrics such as the mean absolute error (MAE), precision, sensitivity, and F1 score. Results showed that the angle-based OD (ABOD) had the best MAE, precision, sensitivity, and F1 score of 8%, 85%, 98%, and 0.90, respectively. The next best-performing OD technique was distance-based OD (DBOD), with MAE, precision, sensitivity, and F1 score of 16%, 71%, 100%, and 0.83, respectively. We tested the ABOD method on several field production datasets by assuming different outlier thresholds (fraction of the data points likely to be outliers). Visual inspection of the processed data showed that the ABOD method effectively identified and removed the outliers from a relatively clean dataset (outlier threshold of 20%) and a highly noisy dataset (outlier threshold of 80%). This algorithm is intuitive and can effectively identify and remove outliers from the field production data to improve production forecasting, reserves estimation, and rate transient analysis for multi-fractured horizontal oil and gas reservoirs.

Adaptivity in continuous massively parallel distance-based outlier detection

Adaptivity in continuous massively parallel distance-based outlier detection

Fast, exact, and parallel-friendly outlier detection algorithms with proximity graph in metric spaces

In many fields, e.g., data mining and machine learning, distance-based outlier detection (DOD) is widely employed to remove noises and find abnormal phenomena, because DOD is unsupervised, can be employed in any metric spaces, and does not have any assumptions of data distributions. Nowadays, data mining and machine learning applications face the challenge of dealing with large datasets, which requires efficient DOD algorithms. We address the DOD problem with two different definitions. Our new idea, which solves the problems, is to exploit an in-memory proximity graph. For each problem, we propose a new algorithm that exploits a proximity graph and analyze an appropriate type of proximity graph for the algorithm. Our empirical study using real datasets confirms that our DOD algorithms are significantly faster than state-of-the-art ones.

Little data is often enough for distance-based outlier detection

Many real-world use cases benefit from fast training and prediction times, and much research went into speeding up distance-based outlier detection methods to millions of data points. Contrary to popular belief, our findings suggest that little data is often enough for distance-based outlier detection models. We show that using only a tiny fraction of the data to train distance-based outlier detection models often leads to no significant reduction in predictive performance and detection variance over a wide range of tabular datasets. Furthermore, we compare a data reduction based on random subsampling and clustering-based prototypes and show that both approaches yield similar outlier detection results. Simple random subsampling, thus, proves to be a useful benchmark and baseline for future research on speeding up distance-based outlier detection.

Explainable Distance-Based Outlier Detection in Data Streams

Explaining outliers is a topic that attracts a lot of interest; however existing proposals focus on the identification of the relevant dimensions. We extend this rationale for unsupervised distance-based outlier detection, and through investigating subspaces, we propose a novel labeling of outliers in a manner that is intuitive for the user and does not require any training at runtime. Moreover, our solution is applicable to online settings and a complete prototype for detecting and explaining outliers in data streams using massive parallelism has been implemented. Our solution is evaluated in terms of both the quality of the labels derived and the performance.

Method of Automatically Detecting the Abnormal First Arrivals Using Delay Time (December 2020)

Abnormal first-arrival times in the automatic first-arrival picking significantly affect the structural inversion and static correction. After implementing an automatic picking, the manual intervention is still required to correct or eliminate false first-arrival times, which seriously affects the efficiency of the first-arrival picking. Therefore, it is necessary to develop automatic removal methods for abnormal first-arrival times. Here, we propose a distance-based outlier detection algorithm to automatically and effectively eliminate the abnormal first-arrival times. In order to identify the abnormal first-arrival times, each first-arrival time is decomposed into a receiver delay time and a shot delay time. Then, according to offset and azimuth information, we eliminate the influences of offset and shot delay times on the first-arrival times in a single shot to get the receiver delay times. We further rearrange the delay times in polar coordinates. Finally, we apply a distance-based outlier detection algorithm to the delay times on different azimuths and delete the first-arrival times that correspond to the detected outliers. Our study indicates that the distance-based outlier detection algorithm can effectively eliminate the abnormal first-arrival times. The results from real data processing demonstrate that the proposed automatic removal approach of the abnormal first-arrival times can effectively improve the quality of the first-arrival times. In addition, we find that our proposed algorithm has a better performance than a least-squares regression method for low signal-to-noise ratio (SNR) seismic data.

Efficient Detection Using Soft Computing Approach of Modified Fuzzy C-Means Based Outlier Detection in Electronics Patient Records Systems

Outliers are the data objects or data elements which are deviated from the observation or population in the dataset. Normally outliers are considered as noise data but in recent days outliers are taken vital role in the applications like healthcare systems to energetically check for any malignant, irregular, or abnormal behaviour. Accuracy of the outlier detection is purely based upon the efficiency of outlier detection methods and application where the outlier detection is involved. In this paper we design hybrid three phase modified fuzzy c-means and diverse distance based outlier detection method for distributed dataset to detect the unusual usage of data and illegitimate approach in large-scale integral networks, especially in healthcare sectors (EPR systems). The proposed algorithm combining the features of modified C-Means Fuzzification, Z-score and Manhattan distance in outlier identification. The proposed algorithm provides efficiency in outlier detection on univariate EPRdata. This paper also mainly focuses the application intrusion detection in healthcare data. The algorithm is tested on real world dataset of machine learning database repository (UCIML).

Uncertain distance-based outlier detection with arbitrarily shaped data objects

Enabling information systems to face anomalies in the presence of uncertainty is a compelling and challenging task. In this work the problem of unsupervised outlier detection in large collections of data objects modeled by means of arbitrary multidimensional probability density functions is considered. We present a novel definition of uncertain distance-based outlier under the attribute level uncertainty model, according to which an uncertain object is an object that always exists but its actual value is modeled by a multivariate pdf. According to this definition an uncertain object is declared to be an outlier on the basis of the expected number of its neighbors in the dataset. To the best of our knowledge this is the first work that considers the unsupervised outlier detection problem on data objects modeled by means of arbitrarily shaped multidimensional distribution functions. We present the UDBOD algorithm which efficiently detects the outliers in an input uncertain dataset by taking advantages of three optimized phases, that are parameter estimation, candidate selection, and the candidate filtering. An experimental campaign is presented, including a sensitivity analysis, a study of the effectiveness of the technique, a comparison with related algorithms, also in presence of high dimensional data, and a discussion about the behavior of our technique in real case scenarios.

Real-time distance-based outlier detection in data streams

Real-time outlier detection in data streams has drawn much attention recently as many applications need to be able to detect abnormal behaviors as soon as they occur. The arrival and departure of streaming data on edge devices impose new challenges to process the data quickly in real-time due to memory and CPU limitations of these devices. Existing methods are slow and not memory efficient as they mostly focus on quick detection of inliers and pay less attention to expediting neighbor searches for outlier candidates. In this study, we propose a new algorithm, CPOD, to improve the efficiency of outlier detections while reducing its memory requirements. CPOD uses a unique data structure called "core point" with multi-distance indexing to both quickly identify inliers and reduce neighbor search spaces for outlier candidates. We show that with six real-world and one synthetic dataset, CPOD is, on average, 10, 19, and 73 times faster than M_MCOD, NETS, and MCOD, respectively, while consuming low memory.

RETRACTED ARTICLE: Intelligent fuzzy rule-based approach with outlier detection for secured routing in WSN

In wireless sensor networks (WSNs), energy optimization and the provision of security are the major design challenges. Since the wireless sensor devices are energy constrained, the issue of high energy consumption by the malicious nodes must be addressed well in order to enhance the network performance by making increased network lifetime, reduced energy consumption and delay. In the past, many researchers worked in the provision of new techniques for providing improved security to WSN in order to enhance the reliability in the routing process. However, most of the existing routing techniques are not able to achieve the required security through the use of intelligent techniques for safeguarding the sensor nodes from malicious attacks. In order to address these problems, a new fuzzy temporal clustering-based secured communication model with trust analysis and outlier detection has been developed in this research work. For this purpose, a new fuzzy temporal rule-based cluster-based routing algorithm with trust modelling and outlier detection for monitoring the nodes participating in the communication has been proposed. In addition, a fuzzy temporal rule- and distance-based outlier detection algorithm is also proposed in this paper for distinguishing the malicious nodes from other nodes within each cluster of the network and has been used in the secured routing algorithm. The proposed secure routing algorithm uses the temporal reasoning tasks of explanation-based learning and prediction as well as spatial constraints for making efficient routing decisions through the application of trust and key management techniques for performing effective authentication of nodes and thereby isolating the malicious nodes from communication through outlier detection. By applying these two proposed algorithms for communication in the proposed work, it is proved through experiments that the proposed secure routing algorithm and the outlier detection algorithm are able to perform secured and reliable routing through genuine cluster head nodes more effectively. Moreover, these two algorithms provide improved quality of service with respect to the reliability of communication, packet delivery ratio, reduction in end-to-end delay and reduced energy consumption.

An Efficient Model for Mining Outlier Opinions

In the internet era, opinion mining became a critical technique used in many applications. The internet offers a featured chance for users to express and share their views and experiences anywhere and at any time through various methods as online reviews, personal blogs, Facebook, Twitter and companies’ websites. Such treasure of online data generated by users play an essential role in decision-making process and have the ability to make radical changes in several fields. Although the opinionated text can provide significantly invaluable information for the wide community either are individuals, business, or government, the outlier or anomaly opinions could have the same impact but in opposite manner which harm these fields. Consequently, there is an urge to develop techniques to detect the outlier opinions and avoid their negative impacts on several application domains which rely on opinion mining. In this paper, an efficient model for mining outlier opinions has been proposed. The proposed MOoM model, stands for Mining Outlier Opinion Model, offers for the first time the ability to mine outlier opinions from product’s free-text reviews. Accordingly, it can help the decision makers to improve the overall sentiment analysis process and perform further analysis on the outlier opinions to get better understanding for them and avoid their negative impact. The proposed model consists of three modules; Data preprocessing module, Opinion mining module, and outlier opinions detection module. The proposed model utilizes the lexicon-based approach to extract sentiment polarity from each review in the dataset. Also, it uses the Distance-based outlier detection algorithm to produce a graded list of review holders with outlier opinions. Experimental study is presented to evaluate the proposed model and the results proved the model’s ability to detect outlier opinions in the product reviews effectively. The model is adaptable to be used in other fields rather than product’s reviews by customizing its modules’ layers.

An efficient outlier detection approach on weighted data stream based on minimal rare pattern mining

The distance-based outlier detection method detects the implied outliers by calculating the distance of the points in the dataset, but the computational complexity is particularly high when processing multidimensional datasets. In addition, the traditional outlier detection method does not consider the frequency of subsets occurrence, thus, the detected outliers do not fit the definition of outliers (i.e., rarely appearing). The pattern mining-based outlier detection approaches have solved this problem, but the importance of each pattern is not taken into account in outlier detection process, so the detected outliers cannot truly reflect some actual situation. Aimed at these problems, a two-phase minimal weighted rare pattern mining-based outlier detection approach, called MWRPM-Outlier, is proposed to effectively detect outliers on the weight data stream. In particular, a method called MWRPM is proposed in the pattern mining phase to fast mine the minimal weighted rare patterns, and then two deviation factors are defined in outlier detection phase to measure the abnormal degree of each transaction on the weight data stream. Experimental results show that the proposed MWRPM-Outlier approach has excellent performance in outlier detection and MWRPM approach outperforms in weighted rare pattern mining.

NETS

This paper addresses the problem of efficiently detecting outliers from a data stream as old data points expire from and new data points enter the window incrementally. The proposed method is based on a newly discovered characteristic of a data stream that the change in the locations of data points in the data space is typically very insignificant. This observation has led to the finding that the existing distance-based outlier detection algorithms perform excessive unnecessary computations that are repetitive and/or canceling out the effects. Thus, in this paper, we propose a novel set-based approach to detecting outliers, whereby data points at similar locations are grouped and the detection of outliers or inliers is handled at the group level. Specifically, a new algorithm NETS is proposed to achieve a remarkable performance improvement by realizing set-based early identification of outliers or inliers and taking advantage of the "net effect" between expired and new data points. Additionally, NETS is capable of achieving the same efficiency even for a high-dimensional data stream through two-level dimensional filtering . Comprehensive experiments using six real-world data streams show 5 to 25 times faster processing time than state-of-the-art algorithms with comparable memory consumption. We assert that NETS opens a new possibility to real-time data stream outlier detection.

Unsupervised Distance-Based Outlier Detection using Reversible KNN with Fuzzy Clustering

Unsupervised Distance-Based Outlier Detection using Reversible KNN with Fuzzy Clustering

An Effective Minimal Probing Approach With Micro-Cluster for Distance-Based Outlier Detection in Data Streams

Outlier detection in data streams is considered a significant task in data mining that targets the discovery of elements in an unprecedented data arrival rate. The fast arrival of data demands fast computation within the shortest period, and with minimal memory usage. Detecting distance-based outliers in such a scenario are more complicated. Existing techniques such as the two best-known methods - Micro-Cluster Outlier Detection (MCOD) and Thresh_LEAP have presented some solutions to these challenges. However, the combination of the strength of both techniques can be a lot more improvement to the individual methods proposed. Therefore, in this paper, we propose a method called Micro-Cluster with Minimal Probing (MCMP), which is a hybrid approach of the combination of the strength of MCOD and Thresh_LEAP. We offer a new distance-based outlier detection technique to minimize the computational cost in detecting distance-based outliers effectively. The proposed MCMP technique is comprised of two approaches. Firstly, we adopt micro-clusters to mitigate the range query search. Then, to deal with the objects outside the micro-clusters, we propose the concept of differentiating between strong and trivial inliers. The proposed method improves the computational speed and memory consumption, while simultaneously maintaining the outlier detection accuracy. Our experiments are conducted on both real-world and synthetic data sets. We varied the window size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(w)$ </tex-math></inline-formula> , neighbor count threshold <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(k)$ </tex-math></inline-formula> and distance threshold <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(R)$ </tex-math></inline-formula> , and observed that our method outperforms the state-of-the-art methods in both CPU time and memory consumption in the majority of the datasets.

Adapted K-Nearest Neighbors for Detecting Anomalies on Spatio–Temporal Traffic Flow

Outlier detection is an extensive research area, which has been intensively studied in several domains such as biological sciences, medical diagnosis, surveillance, and traffic anomaly detection. This paper explores advances in the outlier detection area by finding anomalies in spatio-temporal urban traffic flow. It proposes a new approach by considering the distribution of the flows in a given time interval. The flow distribution probability (FDP) databases are first constructed from the traffic flows by considering both spatial and temporal information. The outlier detection mechanism is then applied to the coming flow distribution probabilities, the inliers are stored to enrich the FDP databases, while the outliers are excluded from the FDP databases. Moreover, a k-nearest neighbor for distance-based outlier detection is investigated and adopted for FDP outlier detection. To validate the proposed framework, real data from Odense traffic flow case are evaluated at ten locations. The results reveal that the proposed framework is able to detect the real distribution of flow outliers. Another experiment has been carried out on Beijing data, the results show that our approach outperforms the baseline algorithms for high-urban traffic flow.