Outlier Detection Techniques Research Articles

A Survey on Cluster Based Outlier Detection Techniques in Data Stream

A Survey on Cluster Based Outlier Detection Techniques in Data Stream

ENHANCEMENT STRATEGIES FOR FRAME-TO-FRAME UAS STEREO VISUAL ODOMETRY

Autonomous navigation of indoor unmanned aircraft systems (UAS) requires accurate pose estimations usually obtained from indirect measurements. Navigation based on inertial measurement units (IMU) is known to be affected by high drift rates. The incorporation of cameras provides complementary information due to the different underlying measurement principle. The scale ambiguity problem for monocular cameras is avoided when a light-weight stereo camera setup is used. However, also frame-to-frame stereo visual odometry (VO) approaches are known to accumulate pose estimation errors over time. Several valuable real-time capable techniques for outlier detection and drift reduction in frame-to-frame VO, for example robust relative orientation estimation using random sample consensus (RANSAC) and bundle adjustment, are available. This study addresses the problem of choosing appropriate VO components. We propose a frame-to-frame stereo VO method based on carefully selected components and parameters. This method is evaluated regarding the impact and value of different outlier detection and drift-reduction strategies, for example keyframe selection and sparse bundle adjustment (SBA), using reference benchmark data as well as own real stereo data. The experimental results demonstrate that our VO method is able to estimate quite accurate trajectories. Feature bucketing and keyframe selection are simple but effective strategies which further improve the VO results. Furthermore, introducing the stereo baseline constraint in pose graph optimization (PGO) leads to significant improvements.

ENHANCEMENT STRATEGIES FOR FRAME-TO-FRAME UAS STEREO VISUAL ODOMETRY

Abstract. Autonomous navigation of indoor unmanned aircraft systems (UAS) requires accurate pose estimations usually obtained from indirect measurements. Navigation based on inertial measurement units (IMU) is known to be affected by high drift rates. The incorporation of cameras provides complementary information due to the different underlying measurement principle. The scale ambiguity problem for monocular cameras is avoided when a light-weight stereo camera setup is used. However, also frame-to-frame stereo visual odometry (VO) approaches are known to accumulate pose estimation errors over time. Several valuable real-time capable techniques for outlier detection and drift reduction in frame-to-frame VO, for example robust relative orientation estimation using random sample consensus (RANSAC) and bundle adjustment, are available. This study addresses the problem of choosing appropriate VO components. We propose a frame-to-frame stereo VO method based on carefully selected components and parameters. This method is evaluated regarding the impact and value of different outlier detection and drift-reduction strategies, for example keyframe selection and sparse bundle adjustment (SBA), using reference benchmark data as well as own real stereo data. The experimental results demonstrate that our VO method is able to estimate quite accurate trajectories. Feature bucketing and keyframe selection are simple but effective strategies which further improve the VO results. Furthermore, introducing the stereo baseline constraint in pose graph optimization (PGO) leads to significant improvements.

Gravitational outlier detection for wireless sensor networks

SummaryAccuracy of sensed data and reliable delivery are the key concerns in addition to several other network‐related issues in wireless sensor networks (WSNs). Early detection of outliers reduces subsequent unwanted transmissions, thus preserving network resources. Recent techniques on outlier detection in WSNs are computationally expensive and based on message exchange. Message exchange‐based techniques incur communication overhead and are less preferred in WSNs. On the other hand, machine learning‐based outlier detection techniques are computationally expensive for resource constraint sensor nodes. The novelty of this paper is that it proposes a simple, non message exchange based, in‐network, real‐time outlier detection algorithm based on Newton's law of gravity. The mechanism is evaluated for its accuracy in detecting outliers, computational cost, and its influence on the network traffic and delay. The outlier detection mechanism resulted in almost 100% detection accuracy. Because the mechanism involves no message exchanges, there is a significant reduction in network traffic, energy consumption and end‐to‐end delay. An extension of the proposed algorithm for transient data sets is proposed, and analytic evaluation justifies that the mechanism is reactive to time series data. Copyright © 2016 John Wiley & Sons, Ltd.

An Efficient Unsupervised Clustered Adaptive Antihub Technique for Outlier Detection in High Dimensional Data

Objective: The objective of this paper is to find the inconsistent objects in data which has high dimension through reduced computation time and increased accuracy. Methods: Hubness specifically Antihubs (points that rarely occur in k nearest neighbor lists) is the newly recognized concept for handling data which has high dimension. The advanced version of Antihub is Antihub2 which is for reconsidering the outlier score of a point obtained by the Antihub method. However, regarding computation time, Antihub2 runs slower. This paper institutes an approach called AdaptiveAntihub2Clust, which is a clustered Adaptive Antihub technique for unsupervised outlier detection to reduce computation time and to improve the accuracy. Findings: The results of an existing Antihub2 method is compared with the proposed method called AdaptiveAntihub2Clust. The experimental results elucidate that AdaptiveAntihub2Clust outperforms well than Antihub2 and also resolved that there is not only a substantial decrease in computation time but also progress in accuracy occurred while the newly built approach is practically used for finding outliers. Applications: The irrelevant objects may ascend due to numerous faults. Detection of such objects identifies the mistakes and fraud before they deteriorate with terrible significances and cleanses the data for further processing.

Spatio-temporal quality control: implications and applications for data consumers and aggregators

Quality control for real-time spatio-temporal data is often presented from the perspective of the original owner and provider of the data, and focuses on general techniques for outlier detection or uses domain-specific knowledge and rules to assess quality. The impact of quality control on the data aggregator and redistributor is neglected. As sensor networks proliferate, multiple providers can distribute and redistribute the same original sensor data. Relationships between providers become complex, with data acquired from original and third-party sources. One provider may acquire data from another, and so forth, resulting in larger data sets with value-added components such as quality indicators, but with costs such as increased lag between original observation times and (re)distribution times. The focus of this paper is to define and demonstrate quality control measures for real-time, spatio-temporal data from the perspective of an aggregator to provide tools for evaluation and comparison of overlapping, real-time, spatio-temporal data providers and for assessment and optimization of data acquisition, system operation and data redistribution. We define simple measures that account for temporal completeness and spatial coverage. The measures and methods developed are tested on real-world data and applications. Our results show that these simple measures combine to form methods that are useful in comparing providers and identifying patterns in data which can then be exploited to optimize system performance relative to bandwidth, and to assess the impact of provider quality control mechanisms. The simple measures presented demonstrate the utility of quantifying data quality from the perspective of the data aggregator and redistributor.

Real valued diffusion-weighted imaging using decorrelated phase filtering.

Because of the intrinsic low signal-to-noise ratio in diffusion-weighted imaging (DWI), magnitude processing often causes an overestimation of the signal's amplitude. This results in low-estimation accuracy of diffusion models and reduced contrast because of a superposition of the image signal and the noise floor. We adopt a new phase correction (PC) technique that yields real valued diffusion data while maintaining a Gaussian noise distribution. We conduct simulations of the noise propagation in the echo-planar imaging reconstruction chain to determine the spatial noise correlation in the image. Using the correlation pattern, optimized filter kernels are derived to estimate the true phase of the signal in each voxel. Furthermore, we adopt an outlier detection technique to replace the real value by the magnitude in case of substantial signal loss resulting from incorrect PC. The benefits of our method are demonstrated on Monte Carlo simulations, DWI data acquired from healthy volunteer experiments, estimated parameters of the diffusion kurtosis imaging model, and the model-free diffusion spectrum imaging technique. The improved PC approach significantly reduces the noise bias and only slightly increases the sensitivity to local phase variations. PC can enhance the usefulness of higher b-values, allowing deeper insights into tissue microstructure. Magn Reson Med 77:559-570, 2017. © 2016 International Society for Magnetic Resonance in Medicine.

Survey on Outlier Detection in Data Stream

mining provides a way for finding hidden and useful knowledge from the large amount of data .usually we find any information by finding normal trends or distribution of data .But sometimes rare event or data object may provide information which is very interesting to us .Outlier detection is one of the task of data mining .It finds abnormal data point or sequence hidden in the dataset .Data stream is unbounded sequence of data with explicit or implicit temporal context .Data stream is uncertain and dynamic in nature. Traditional outlier detection techniques for static data which require whole dataset for modelling is not suitable for data stream because whole data stream cannot be stored. Network intrusion detection ,web click stream analysis ,fraud detection ,fault detection in machines ,sensor data analysis are some of the applications of data stream outlier detection .In this paper, we have described several issues in data stream outlier detection and usual approaches or techniques for finding outlier in data stream .

Automated Algorithmic Error Resilience Based on Outlier Detection

The authors propose automated algorithmic error resilience based on outlier detection. The approach exploits characteristic behavior of a class of applications to create metric functions that normally produce values according to a designed distribution or behavior and produce outlier values when computations are affected by errors. For a robust algorithm that employs such an approach, error detection becomes equivalent to outlier detection. As such, the authors use well-established, statistically rigorous techniques for outlier detection to effectively and efficiently detect errors, and subsequently correct them. The authors' error-resilient algorithms incur significantly lower overhead than traditional hardware and software error-resilience techniques (such as triple modular redundancy). In addition, compared to previous approaches to application-based error resilience, the authors' approaches parameterize the robustification process, making it easy to automatically transform large classes of applications into robust applications with the use of parser-based tools and minimal programmer effort. They demonstrate the use of automated error resilience based on outlier detection for dynamic programming problems. For error rates up to 10E-3, the error-resilient algorithms achieve the same output quality as their error-free counterparts with significantly lower overhead (less than 59 percent for monadic problems and 263 percent for polyadic problems, on average) than conventional hardware and software error-resilience techniques.

Stereo-Based Visual Odometry for Autonomous Robot Navigation

Mobile robots should possess accurate self-localization capabilities in order to be successfully deployed in their environment. A solution to this challenge may be derived from visual odometry (VO), which is responsible for estimating the robot's pose by analysing a sequence of images. The present paper proposes an accurate, computationally-efficient VO algorithm relying solely on stereo vision images as inputs. The contribution of this work is twofold. Firstly, it suggests a non-iterative outlier detection technique capable of efficiently discarding the outliers of matched features. Secondly, it introduces a hierarchical motion estimation approach that produces refinements to the global position and orientation for each successive step. Moreover, for each subordinate module of the proposed VO algorithm, custom non-iterative solutions have been adopted. The accuracy of the proposed system has been evaluated and compared with competent VO methods along DGPS-assessed benchmark routes. Experimental results of relevance to rough terrain routes, including both simulated and real outdoors data, exhibit remarkable accuracy, with positioning errors lower than 2%.

Smart outlier detection of wireless sensor network

Data sets collected from wireless sensor networks (WSN) are usually considered unreliable and subject to errors due to limited sensor capabilities and hard environment resulting in a subset of the sensors data called outlier data. This paper proposes a technique to detect outlier data base on spatial-temporal similarity among data collected by geographically distributed sensors. The proposed technique is able to identify an abnormal subset of data collected by sensor node as outlier data. Moreover, the proposed technique is able to classify this abnormal observation, an error data set or event affected set. Simulation result shows that high detection rate is achieved compared to conventional outlier detection techniques while preserving low positive false alarm rate.

Outlier Detection Techniques for Localization in Wireless Sensor Networks: A Survey

In wireless sensor networks (WSNs), localization is one of the most important topics because the location information is typically useful for many applications. The primary data used in a localization process include the locations of anchor nodes and the distances between neighboring nodes. However, these data may contain outliers that deviate from their true values. The existence of the outliers might make the estimated positions not accurate. Thus, it is important to detect and handle outliers in order to achieve high localization accuracy. In this paper, we survey the existing outlier detection techniques for localization in wireless sensor networks. We provide taxonomy for classifying outlier detection techniques in WSNs localization based on different features. In addition, we present comparisons of these techniques. Finally, we discuss the future research directions in this area.

An Approach to Perform Uncertainity Analysis on a Spatial Dataset using Clustering and Distance based Outlier Detection Technique

Background: In past years, many methods have been implemented for maintaining and supervising uncertain data that may occur due to collection of data in new ways which results in missing values, erroneous data. The main aim of this work is to help the end user to get correct information about spatial data. Method: The behaviour of data as an outlier is the result of uncertainty. The challenge in spatial data sets is to cluster uncertain objects. Hence, unsupervised clustering can be used to deal with this type of data. In this paper, the difficulty of outlier detection with uncertain data is examined. Finding: To improve the performance and quality, Voronoi Diagram is used which partition the objects into each cell and helps to see the exact location of an object. The integral part is the pre-processing step of removing uncertainty to avoid wrong interpretation. Furthermore, CLARA (Clustering LARge Applications) algorithm is applied to produce the high quality clusters. It has an in-built function of outlier detection too and it is suitable for large data set. This algorithm uses Mahalanobis Distance to calculate the distance between cluster and its members, to remove outliers and reduce uncertainty for feasible and supporting inputs. This procedure can be a valid provision to be use in geo-database creation. Improvement: The methodology can be enhanced by designing the procedure to develop a Decision Support System (DSS) for spatial database creation.

Initialization of K-modes clustering using outlier detection techniques

The K-modes clustering has received much attention, since it works well for categorical data sets. However, the performance of K-modes clustering is especially sensitive to the selection of initial cluster centers. Therefore, choosing the proper initial cluster centers is a key step for K-modes clustering. In this paper, we consider the initialization of K-modes clustering from the view of outlier detection. We present two different initialization algorithms for K-modes clustering, where the first is based on the traditional distance-based outlier detection technique, and the second is based on the partition entropy-based outlier detection technique. By using the above two outlier detection techniques to calculate the degree of outlierness of each object, our algorithms can guarantee that the chosen initial cluster centers are not outliers. Moreover, during the process of initialization, we adopt a new distance metric — weighted matching distance metric, to calculate the distance between two objects described by categorical attributes. Experimental results on several UCI data sets demonstrate the effectiveness of our initialization algorithms for K-modes clustering.

Hubness in Unsupervised Outlier Detection Techniques for High Dimensional Data –A Survey

Hubness in Unsupervised Outlier Detection Techniques for High Dimensional Data –A Survey

Ranking Fragment Ions Based on Outlier Detection for Improved Label-Free Quantification in Data-Independent Acquisition LC-MS/MS.

Data-independent acquisition LC-MS/MS techniques complement supervised methods for peptide quantification. However, due to the wide precursor isolation windows, these techniques are prone to interference at the fragment ion level, which, in turn, is detrimental for accurate quantification. The nonoutlier fragment ion (NOFI) ranking algorithm has been developed to assign low priority to fragment ions affected by interference. By using the optimal subset of high-priority fragment ions, these interfered fragment ions are effectively excluded from quantification. NOFI represents each fragment ion as a vector of four dimensions related to chromatographic and MS fragmentation attributes and applies multivariate outlier detection techniques. Benchmarking conducted on a well-defined quantitative data set (i.e., the SWATH Gold Standard) indicates that NOFI on average is able to accurately quantify 11-25% more peptides than the commonly used Top-N library intensity ranking method. The sum of the area of the Top3-5 NOFIs produces similar coefficients of variation as compared to that with the library intensity method but with more accurate quantification results. On a biologically relevant human dendritic cell digest data set, NOFI properly assigns low-priority ranks to 85% of annotated interferences, resulting in sensitivity values between 0.92 and 0.80, against 0.76 for the Spectronaut interference detection algorithm.

Ranking Optimization using Outlier Detection Technique

Product ranking optimization is an emerging required research area where we are getting a heavy duly competition day by day, number of products with sort of ranges are available in market, finding the best out of the different product is a major problem, several times it observed that number of survey has been made in order to make a product ranking and show it to users about the best available product for their requirement in the market, the ranking been held on multiple required attributes work or play the role to increase or decrease their ranking. Existing Line-up technique is only provided rank to number of products. It did not optimize the ranking. Now, in this paper, defeat these entire problems and provide the best solution. Introduce the outlier detection technique to optimize the product ranking according to requirement feature or attribute. Here i am going to implement some algorithm to detect best outliers such as distributing solving set algorithm, line-up algorithm & improved LAZY DSS.

A computationally efficient order statistics based outlier detection technique for EEG signals.

Detecting artifacts in EEG data produced by muscle activity, eye blinks and electrical noise is a common and important problem in EEG applications. We present a novel outlier detection method based on order statistics. We propose a 2 step procedure comprising of detecting noisy EEG channels followed by detection of noisy epochs in the outlier channels. The performance of our method is tested systematically using simulated and real EEG data. Our technique produces significant improvement in detecting EEG artifacts over state-of-the-art outlier detection technique used in EEG applications. The proposed method can serve as a general outlier detection tool for different types of noisy signals.

Spotting Outliers in Large Distributed Datasets using Cell Density based Approach

Outliers are abnormal instances or observations. Detecting data outliers is a very important concept in Knowledge data discovery. Outlier detection has been studied in the context of a large number of research areas like large distributed systems, data mining, wireless sensor networks(WSN), health monitoring, environmental science, statistics, etc., Density based (DB) outlier detection techniques are robust in detecting outliers. In many applications, too much voluminous distributed data is generating every day. Finding deviating observations in the large distributed database rather than in any individual database is not a simple task. Integrating distributed database cause two major problems. First, render massive data from different databases. In addition, data integration may cause violation of data security and leakage of sensitive information. In this work we propose cell density based mechanism for outlier detection (CDOD) in large distributed databases. A centralized detection paradigm is used; it allows overcoming the expensive data integration and information leakage. The experimental results show robustness for finding outliers in large number of databases, instances and attributes.

Outlier Detection in Dataset using Hybrid Approach

is a data point that deviates too much from the rest of dataset. Most of real-world dataset have outlier. Outlier analysis is one of the techniques in data mining whose task is to discover the data which have an exceptional behavior compare to remaining dataset. Outlier detection plays an important role in data mining field. Outlier Detection is useful in many fields like Medical, Network intrusion detection, Credit card fraud detection, medical, fault diagnosis in machines, etc. In order to deal with outlier, clustering method is used. Outlier detection contains clustering and finding outlier by applying any outlier detection technique. For that K- mean is widely used to cluster the dataset. Different techniques like statistical-based, distance-based, and deviation-based and density based methods are used to detect outlier. The experiment result shows that existing algorithm perform better than proposed cluster-based and distance-based Algorithm.