Local Distance-based Outlier Factor Research Articles

Efficient distributed privacy-preserving collaborative outlier detection

As a common way to identify anomalous data, outlier detection is widely applicable for intrusions detection, adverse reactions analysis, financial fraud prevention, etc. The accuracy of outlier detection depends crucially on the number of data involved in the test, i.e., the more data participate in detection, the higher accuracy we get. For this reason, cross-dataset collaborative outlier detection is introduced to conquer the lack of data in a single-dataset setting. However, privacy concerns seriously prevent the application of collaborative outlier detection, since most organization are unwilling to share their data with others directly in practice. In this paper, we present efficient protocols for privacy preserving collaborative outlier detection from arbitrarily partitioned data using Local Distance-based Outlier Factor (LDOF). Our protocols fall in the two-server model where data owners distribute their private data among two non-colluding servers who detect outlier on the joint data by secure two-party computation. In particular, we perform arithmetic operations which takes place inside LDOF on arithmetic circuits instead of boolean circuits, and perform sorting operations on boolean circuits. Such a design enables standard operations are performed with suitable circuits, and thus our scheme is more efficient. In addition, to further improve protocol efficiency, local sensitive hash (LSH) is utilized to filter out data which do not need secure computation to reduce the the amount of shared data. We implement our system in C++ on real data. The security analysis and experiments show the security and efficiency of the proposed scheme. Our protocols are more faster than the state of previous methods.

Automatic Date Fruit Recognition Using Outlier Detection Techniques and Gaussian Mixture Models

In this paper, we propose a method for automatically recognizing different date varieties. The presence of outlier samples could significantly degrade the recognition outcomes. Therefore, we separately prune samples of each variety from outliers using the Pruning Local Distance-based Outlier Factor (PLDOF) method. Samples of the same variety could have several visual appearances because of the noticeable variation in terms of their visual characteristics. Thus, in order to take this intra-variation into account, we model each variety with a Gaussian Mixture Model (GMM), where each component within the GMM corresponds to one visual appearance. Expectation-Maximization (EM) algorithm was used for parameters estimation and Davies-Bouldin index was used to automatically and precisely estimate the number of components (i.e., appearances). Compared to the related studies, the proposed method 1) is capable to recognize samples though the noticeable variation, in terms of maturity stage and hardness degree, within some varieties; 2) achieves a high recognition rate in spite of the presence of outlier samples; 3) is capable to distinguish between the highly confusing varieties; 4) is fully automatic, as it does not require neither physical measurements nor human assistance. For testing purposes, we introduce a new benchmark which includes the highest number of varieties (11) compared to the previous studies. Experiments show that our method has significantly outperformed several methods, where a high recognition rate of 97.8% has been reached.

Local outlier detection based on information entropy weighting

As a key research area in data mining technologies, outlier detection can expose data inconsistent with the majority in the dataset and therefore is applicable in extensive areas. The addition of entropy weighting to the spatial local outlier measure (SLOM) and local distance-based outlier factor (LDOF) algorithms in outlier data mining, i.e., the adoption of entropy in the calculation of weighted distance is taken into consideration, leads to enhanced accuracy of outlier detection and produces more expense of time. The algorithm of entropy-weighted LDOF is more optimised than that of entropy-weighted SLOM in terms of detection accuracy. The superiority of the entropy-weighted algorithm is verified through experimental results.

Outlier Detection in Ocean Wave Measurements by Using Unsupervised Data Mining Methods

Abstract Outliers are considerably inconsistent and exceptional objects in the data set that do not adapt to expected normal condition. An outlier in wave measurements may be due to experimental and configuration errors, technical defects in equipment, variability in the measurement conditions, rare or unknown conditions such as tsunami, windstorm and etc. To improve the accuracy and reliability of an built ocean wave model, or to extract important and valuable information from collected wave data, detecting of outlying observations in wave measurements is very important. In this study, three typical outlier detection algorithms:Box-plot (BP), Local Distance-based Outlier Factor (LDOF), and Local Outlier Factor (LOF) methods are used to detect outliers in significant wave height (Hs) records. The historical wave data are taken from National Data Buoy Center (NDBC). Finally, those data points are considered as outlier identified by at least two methods which are presented and discussed. Then, Hs prediction has been modelled with and without the presence of outliers by using Regression trees (RTs).

Autonomous selection of i-vectors for PLDA modelling in speaker verification

Recently, systems combining i-vector and probabilistic linear discriminant analysis (PLDA) have become one of the state-of-the-art methods in text-independent speaker verification. The training data of a PLDA model is often collected from a large, diverse population. However, including irrelevant or noisy training data may deteriorate the verification performance. In this paper, we first show that data selection using k-NN improves the speaker verification performance. We then present a robust way of selecting k based on the local distance-based outlier factor (LDOF). We call this method flexible k-NN (fk-NN). We conduct experiments on male and female trials of several telephone conditions of the NIST 2006, 2008, 2010 and 2012 Speaker Recognition Evaluations (SRE). By using fk-NN, we discard a substantial amount of irrelevant or noisy training data without depending on tuning k, and achieve significant performance improvements on the NIST SRE sets.

English

2 ABSTRACT: Outlier detection is the process of finding outlying pattern from a given dataset. Outlier detection became important subject in different knowledge domains. Data size is getting doubled every years there is a need to detect outliers in large datasets as early as possible. In high-dimensional data outlier detection presents various challenges because of curse of dimensionality. By examining again the notion of reverse nearest neighbors in the unsupervised outlier-detection context, high dimensionality can have a different impact. In high dimensions it was observed that the distribution of points in reverse-neighbor counts becomes skewed .This proposed work aims at developing and comparing some of the unsupervised outlier detection methods and propose a way to improve them. This proposed work goes in details about the development and analysis of outlier detection algorithms such as Local Outlier Factor(LOF), Local Distance-Based Outlier Factor(LDOF) , Influenced Outliers and .The concepts of these methods are then combined to implement a new method with distributed approach which improves the results of the previous mentioned ones with reference to speed, complexity and accuracy.

Local Distance-based Outlier Removal for Scattered data through Minimum Spanning Tree

Minimum spanning tree-based algorithm is capable of detecting outlier from data set having irregular shape, size and boundaries. Detecting outliers using Minimum Spanning Tree-based algorithm is a big desire. Outlier detection is an extremely important task in a wide variety of application. Outliers have little or no influence, and may be isolated as noise in the data set. Existing outlier detection methods are ineffective on scattered data set due to implicit data patterns and parameter setting issues. In this paper we define a novel Local Distance-based Outlier Factor (LDOF) to measure the outlier-ness of objects in the scattered data sets. Based on the LDOF computed, we propose a new method to detect outliers present in the scattered data set.

Cluster based Outlier Detection

detection is a fundamental issue in data mining, specifically it has been used to detect and remove anomalous objects from data.mining. The proposed approach to detect outlier includes three methods which are clustering, pruning and computing outlier score. For clustering k-means algorithm is used which partition the dataset into given number of clusters. In pruning, based on some distance measure, points which are closed to centroid of each cluster are pruned. For the unpruned points, local distance based outlier factor (LDOF) measure is calculated. A measure called LDOF, tells how much a point is deviating from its neighbors. The high LDOF value of a point indicates that the point is deviating more from its neighbors and probably it may be an outlier. General Terms Outlier, cluster, pruning, outlier score, k nearest neighbor KeywordsLDOF