Definition Of Outlier Research Articles

Identification of Outliers in Data Envelopment Analysis

Data Envelopment Analysis (DEA) is a deterministic method for the aggregation of multidimensional measures and subsequent efficiency analysis. Due to its inherent determinism, however, it reacts sensitively to outliers in datasets. Existing methods for identifying such outliers have two main disadvantages. First, from a more conceptional point of view, a uniform definition of an outlier is missing. Second, there are technical disadvantages of each method. For instance, arbitrarily limited values have to be set by the user, like the amount of efficiency value from which on a decision making unit is regarded as an outlier. This paper initially presents a definition of outliers, which explicitly takes the specifics of DEA into account. Based on this definition, an approach for identifying outliers in DEA is introduced which explicitly tackles the technical disadvantages and takes them into account in the developed algorithm. The plausibility of this approach is validated on the basis of empirical examples from performance measurement at the university level.

How to Classify, Detect, and Manage Univariate and Multivariate Outliers, With Emphasis on Pre-Registration

Researchers often lack knowledge about how to deal with outliers when analyzing their data. Even more frequently, researchers do not pre-specify how they plan to manage outliers. In this paper we aim to improve research practices by outlining what you need to know about outliers. We start by providing a functional definition of outliers. We then lay down an appropriate nomenclature/classification of outliers. This nomenclature is used to understand what kinds of outliers can be encountered and serves as a guideline to make appropriate decisions regarding the conservation, deletion, or recoding of outliers. These decisions might impact the validity of statistical inferences as well as the reproducibility of our experiments. To be able to make informed decisions about outliers you first need proper detection tools. We remind readers why the most common outlier detection methods are problematic and recommend the use of the median absolute deviation to detect univariate outliers, and of the Mahalanobis-MCD distance to detect multivariate outliers. An R package was created that can be used to easily perform these detection tests. Finally, we promote the use of pre-registration to avoid flexibility in data analysis when handling outliers.

ORE identifies extreme expression effects enriched for rare variants.

Non-coding rare variants (RVs) may contribute to Mendelian disorders but have been challenging to study due to small sample sizes, genetic heterogeneity and uncertainty about relevant non-coding features. Previous studies identified RVs associated with expression outliers, but varying outlier definitions were employed and no comprehensive open-source software was developed. We developed Outlier-RV Enrichment (ORE) to identify biologically-meaningful non-coding RVs. We implemented ORE combining whole-genome sequencing and cardiac RNAseq from congenital heart defect patients from the Pediatric Cardiac Genomics Consortium and deceased adults from Genotype-Tissue Expression. Use of rank-based outliers maximized sensitivity while a most extreme outlier approach maximized specificity. Rarer variants had stronger associations, suggesting they are under negative selective pressure and providing a basis for investigating their contribution to Mendelian disorders. ORE, source code, and documentation are available at https://pypi.python.org/pypi/ore under the MIT license. Supplementary data are available at Bioinformatics online.

Not so implausible: impact of longitudinal assessment of implausible anthropometric measures on obesity prevalence and weight change in children and adolescents

PurposeImplausible anthropometric measures are typically identified using population outlier definitions, conflating implausible and extreme measures. We determined the impact of a longitudinal outlier approach on prevalence of body mass index (BMI) categories and mean change in anthropometric measures in pediatric electronic health record data. MethodsWe examined 996,131 observations from 147,375 children (10–18 years) in the ADVANCE Clinical Data Research Network, a national network of community health centers. Sex-stratified, mixed effects, linear spline regression modeled weight, height, and BMI as a function of age. Longitudinal outliers were defined as observations with studentized residual greater than |6|; population outliers were defined by Centers for Disease Control-defined z-score thresholds. ResultsAt least 99.7% of anthropometric measures were not extreme by longitudinal or population definitions (agreement ≥ 0.995). BMI category prevalence after excluding longitudinal or population outliers differed by less than 0.1%. Among children greater than 85th percentile at baseline, annual mean changes in anthropometric measures were larger in data that excluded longitudinal (girls: 1.24 inches, 12.39 pounds, 1.53 kg/m2; boys: 2.34, 14.08, 1.07) versus population outliers (girls: 0.61 inches, 8.22 pounds, 0.75 kg/m2; boys: 1.53, 11.61, 0.48). ConclusionsLongitudinal outlier methods may reduce underestimation of anthropometric change in children with elevated baseline values.

Understanding Significance Tests From a Non-Mixing Markov Chain for Partisan Gerrymandering Claims

ABSTRACTRecently, Chikina, Frieze, and Pegden proposed a way to assess significance in a Markov chain without requiring that Markov chain to mix. They presented their theorem as a rigorous test for partisan gerrymandering. We clarify that their ε-outlier test is distinct from a traditional global outlier test and does not indicate, as they imply, that a particular electoral map is associated with an extreme level of “partisan unfairness.” In fact, a map could simultaneously be an ε-outlier and have a typical partisan fairness value. That is, their test identifies local outliers but has no power for assessing whether that local outlier is a global outlier. How their specific definition of local outlier is related to a legal gerrymandering claim is unclear given Supreme Court precedent.

The effects of handling outliers on the performance of bankruptcy prediction models

Ratio type financial indicators are the most popular explanatory variables in bankruptcy prediction models. These measures often exhibit heavily skewed distribution because of the presence of outliers. In the absence of clear definition of outliers, ad hoc approaches can be found in the literature for identifying and handling extreme values. However, it is not clear how these different approaches can affect the predictive power of models. There seems to be consensus in the literature on the necessity of handling outliers, at the same time, it is not clear how to define extreme values to be handled in order to maximize the predictive power of models. There are two possible ways to reduce the bias originating from outliers: omission and winsorization. Since the first approach has been examined previously in the literature, we turn our attention to the latter. We applied the most popular classification methodologies in this field: discriminant analysis, logistic regression, decision trees (CHAID and CART) and neural networks (multilayer perceptron). We assessed the predictive power of models in the framework of tenfold stratified crossvalidation and area under the ROC curve. We analyzed the effect of winsorization at 1, 3 and 5% and at 2 and 3 standard deviations, furthermore we discretized the range of each variable by the CHAID method and used the ordinal measures so obtained instead of the original financial ratios. We found that this latter data preprocessing approach is the most effective in the case of our dataset. In order to check the robustness of our results, we carried out the same empirical research on the publicly available Polish bankruptcy dataset from the UCI Machine Learning Repository. We obtained very similar results on both datasets, which indicates that the CHAID-based categorization of financial ratios is an effective way of handling outliers with respect to the predictive performance of bankruptcy prediction models.

A new test of discordancy in cylindrical data

ABSTRACTCylindrical data are bivariate data from the combination of circular and linear variables. However, up to now no work has been done on the detection of outlier in cylindrical data. We introduce a definition of outlier for cylindrical data and present a new test of discordancy to detect outlier in this type of data, based on the k-nearest neighbor’s distance. Cut-off points of the new test statistic based on the Johnson-Wehrly distribution are calculated and its performance is examined using simulation. A practical example is presented using wind speed and wind direction data obtained from the Malaysian Meteorological Department.

Omnibus outlier detection in sensor networks using windowed locality sensitive hashing

Wireless Sensor Networks (WSNs) have become an integral part of cutting edge technological paradigms such as the Internet-of-Things (IoT) which incorporates a variety of smart application scenarios. WSNs include tiny sensors (motes), with constrained hardware capabilities and limited power supply that can collaboratively function in an unsupervised manner for a long period of time. Their purpose is to continuously monitor quantities of interest and provide answers to application queries. Sensor data streams are inherently spatiotemporal in nature, both because mote measurements form multidimensional time series and due to the spatial reference on the data based on the realm sensed by a mote. Motes are designed to be inexpensive, and thus sensory hardware is prone to temporary or permanent failures yielding faulty measurements. Such measurements may unpredictably forge a query answer, while truthful but abnormal mote samples may indicate undergoing phenomena. Therefore, outlier detection in sensor networks is of utmost importance.With limited power supply and communication being by far the main culprit in energy drain, outlier detection techniques in WSNs should achieve appropriate balance between reducing communication and providing real-time, continuously updated outlier reports. Prior works employ probabilistic or best effort approaches to accomplish the task, which either unpredictably compromise outlier detection accuracy or fail to explicitly tune the amount of communicated data. In this work, we introduce an omnibus outlier detection solution over spatiotemporally referenced sensor data that is capable of: (a) directly trading communication reduction for outlier detection quality with predictable accuracy guarantees, (b) accommodating both uni- and multi-dimensional outlier definitions, (c) operating under various streaming window models and (d) incorporating a wide variety of similarity measures to judge outliers.

A simple transformation independent method for outlier definition

Definition and elimination of outliers is a key element for medical laboratories establishing or verifying reference intervals (RIs). Especially as inclusion of just a few outlying observations may seriously affect the determination of the reference limits. Many methods have been developed for definition of outliers. Several of these methods are developed for the normal distribution and often data require transformation before outlier elimination. We have developed a non-parametric transformation independent outlier definition. The new method relies on drawing reproducible histograms. This is done by using defined bin sizes above and below the median. The method is compared to the method recommended by CLSI/IFCC, which uses Box-Cox transformation (BCT) and Tukey's fences for outlier definition. The comparison is done on eight simulated distributions and an indirect clinical datasets. The comparison on simulated distributions shows that without outliers added the recommended method in general defines fewer outliers. However, when outliers are added on one side the proposed method often produces better results. With outliers on both sides the methods are equally good. Furthermore, it is found that the presence of outliers affects the BCT, and subsequently affects the determined limits of current recommended methods. This is especially seen in skewed distributions. The proposed outlier definition reproduced current RI limits on clinical data containing outliers. We find our simple transformation independent outlier detection method as good as or better than the currently recommended methods.

Feature Grouping-Based Outlier Detection Upon Streaming Trajectories

Outlier detection acts as one of the most important analysis tasks for trajectory stream. In stream scenarios, such properties as unlimitedness, time-varying evolutionary, sparsity, and skewness distribution of trajectories pose new challenges to outlier detection technique. Trajectory outlier detection techniques mainly focus on finding trajectory that is dissimilar to the majority of the others, which is based on the hypothesis that they are probably generated by a different mechanism. Most distance-based methods tend to utilize a function (e.g., weighted linear sum) to measure the similarity of two arbitrary objects provided that representative features have been extracted in advance. However, this kind of method is not tailored to identify the outlier which is close to its neighbors according to some features, but behaves significantly different from its neighbors in terms of the other features. To address this issue, we propose a feature grouping-based mechanism that divides all the features into two groups, where the first group ( Similarity Feature ) is used to find close neighbors and the second group ( Difference Feature ) is used to find outliers within the similar neighborhood. According to the feature differences among local adjacent objects in one or more time intervals, we present two outlier definitions, including local anomaly trajectory fragment ( TF-outlier ) and evolutionary anomaly moving object ( MO-outlier ). We devise a basic solution and then an optimized algorithm to detect both types of outliers. Experimental results show that our proposal is both effective and efficient to detect outliers upon trajectory data streams.

Investigating the Adequacy of Response Time Outlier Definitions in Computer-Based Web Surveys Using Paradata SurveyFocus

Web surveys are commonly used in social research because they are usually cheaper, faster, and simpler to conduct than other modes. They also enable researchers to capture paradata such as response times. Particularly, the determination of proper values to define outliers in response time analyses has proven to be an intricate challenge. In fact, to a certain degree, researchers determine them arbitrarily. In this study, we use “SurveyFocus (SF)”—a paradata tool that records the activity of the web-survey pages—to assess outlier definitions based on response time distributions. Our analyses reveal that these common procedures provide relatively sufficient results. However, they are unable to detect all respondents who temporarily leave the survey, causing bias in the response times. Therefore, we recommend a two-step procedure consisting of the utilization of SF and a common outlier definition to attain a more appropriate analysis and interpretation of response times.

Outlier Detection over Massive-Scale Trajectory Streams

The detection of abnormal moving objects over high-volume trajectory streams is critical for real-time applications ranging from military surveillance to transportation management. Yet this outlier detection problem, especially along both the spatial and temporal dimensions, remains largely unexplored. In this work, we propose a rich taxonomy of novel classes of neighbor-based trajectory outlier definitions that model the anomalous behavior of moving objects for a large range of real-time applications. Our theoretical analysis and empirical study on two real-world datasets—the Beijing Taxi trajectory data and the Ground Moving Target Indicator data stream—and one generated Moving Objects dataset demonstrate the effectiveness of our taxonomy in effectively capturing different types of abnormal moving objects. Furthermore, we propose a general strategy for efficiently detecting these new outlier classes called the <underline>m</underline>inimal <underline>ex</underline>amination (MEX) framework. The MEX framework features three core optimization principles, which leverage spatiotemporal as well as the predictability properties of the neighbor evidence to minimize the detection costs. Based on this foundation, we design algorithms that detect the outliers based on these classes of new outlier semantics that successfully leverage our optimization principles. Our comprehensive experimental study demonstrates that our proposed MEX strategy drives the detection costs 100-fold down into the practical realm for applications that analyze high-volume trajectory streams in near real time.

Abstract 176: Door to Needle Targets in the Real World: A Dynamic Statistical Tool for Institutional Stroke Code Metrics

Background: The benefit of tissue plasminogen activator (tPA) in the treatment of acute ischemic stroke is time-dependent. AHA/ASA guidelines recommend a goal door-to-needle (DTN) time of 60 minutes or less. In practice, medically complicated patients can pose challenges leading to prolonged treatment times that skew institutional mean DTN values. We aimed to produce a simple statistical tool to calculate individualized DTN targets while accounting for outliers to help institutions achieve a mean DTN < 60 minutes. Methods: An IRB approved, single-center, retrospective analysis of consecutive acute stroke code case data from Duke University Hospital (Durham, NC) between January 2014 and July 2016 was conducted. We devised a formula (Figure 1) to calculate the goal DTN time (t g ) necessary to achieve an overall mean DTN of < 60 minutes while adjusting for the frequency and magnitude of outliers. This formula was created as a modifiable tool using an individual institution’s outlier definition, current outlier frequency (f m ), and outlier DTN average (t m ) over a pre-defined time period. As proof of concept, the formula was tested using an outlier definition of ≥100 minutes. The formula was modeled using both real-world institutional data and a simulated sample of 200 patients. Results: In 1069 consecutive stroke codes, tPA was administered in 135 cases with mean DTN 68.2 minutes (range 15-205). Of these, 18 cases (13.3%) met our definition of outlier, with t m of 134 minutes. Using the developed formula with institutional data resulted in a target DTN of 48.6 minutes for non-outlier cases to achieve overall mean DTN < 60 minutes. Using the simulated dataset, our tool similarly calculated a goal DTN of 49 minutes. Conclusions: This simple statistical tool is a novel solution for generating an institution-specific goal DTN based on unique organizational data, experiences, and nuances in order to meet the recommended mean DTN. Additionally, these individualized goals may further motivate stroke teams to carry out more efficient clinical care rather than consistently aiming for DTN values < 60 minutes. The tool presented here is easily adaptable to future changes in recommended target goals and can be seamlessly incorporated into institution-wide systems planning.

A hybrid algorithm for mining local outliers in categorical data

Outlier detection is an important task in data mining. Many approaches have been developed to detect outliers. However, most researches focus on global outlier detection. In many situations, the local outlier detection is more valuable than the global outlier detection. In this paper, the existing methods for outlier detection are discussed firstly, and then the definition of local outlier and some formulas are given. Also a hybrid algorithm for mining local outlier is proposed which is based on clustering algorithm and standard deviation in statistics. By calculating the standard deviation of a cluster and local outlier factor of an object in the cluster, we can identify that the clusters with higher standard deviation may have outliers, and the objects with higher local outlier factor can be recognised as outliers. Experimental results on real datasets show that the proposed algorithm is correct and effective for mining local outliers.

AN OPTIMISED SYSTEM FOR GENERATING MULTI-RESOLUTION DTMS USING NASA MRO DATASETS

Within the EU FP-7 iMars project, a fully automated multi-resolution DTM processing chain, called Co-registration ASP-Gotcha Optimised (CASP-GO) has been developed, based on the open source NASA Ames Stereo Pipeline (ASP). CASP-GO includes tiepoint based multi-resolution image co-registration and an adaptive least squares correlation-based sub-pixel refinement method called Gotcha. The implemented system guarantees global geo-referencing compliance with respect to HRSC (and thence to MOLA), provides refined stereo matching completeness and accuracy based on the ASP normalised cross-correlation. We summarise issues discovered from experimenting with the use of the open-source ASP DTM processing chain and introduce our new working solutions. These issues include global co-registration accuracy, de-noising, dealing with failure in matching, matching confidence estimation, outlier definition and rejection scheme, various DTM artefacts, uncertainty estimation, and quality-efficiency trade-offs.

AN OPTIMISED SYSTEM FOR GENERATING MULTI-RESOLUTION DTMS USING NASA MRO DATASETS

Abstract. Within the EU FP-7 iMars project, a fully automated multi-resolution DTM processing chain, called Co-registration ASP-Gotcha Optimised (CASP-GO) has been developed, based on the open source NASA Ames Stereo Pipeline (ASP). CASP-GO includes tiepoint based multi-resolution image co-registration and an adaptive least squares correlation-based sub-pixel refinement method called Gotcha. The implemented system guarantees global geo-referencing compliance with respect to HRSC (and thence to MOLA), provides refined stereo matching completeness and accuracy based on the ASP normalised cross-correlation. We summarise issues discovered from experimenting with the use of the open-source ASP DTM processing chain and introduce our new working solutions. These issues include global co-registration accuracy, de-noising, dealing with failure in matching, matching confidence estimation, outlier definition and rejection scheme, various DTM artefacts, uncertainty estimation, and quality-efficiency trade-offs.

Outlier Detection for Multivariate Calibration in Near Infrared Spectroscopic Analysis by Model Diagnostics

Abstract Outlier detection is an important task in multivariate calibration because the quality of a calibration model is determined by that of the calibration data. An outlier detection method was proposed for near infrared (NIR) spectral analysis. The method was based on the definition of outlier and the principle of partial least squares (PLS) regression, i.e., an outlier in a dataset behaved differently from the rest, and the prediction result of a PLS model was an accumulation of several independent latent variables. Therefore, the proposed method built a PLS model with a calibration dataset, and then the contribution of each latent variable was investigated. Outliers were detected by comparing these contributions. An NIR spectral dataset of orange juice samples was adopted for testing the method. Six outliers were detected in the calibration set. The root mean squared error of cross validation (RMSECV) was reduced from 16.870 to 4.809 and the root mean squared error of prediction (RMSEP) was reduced from 3.688 to 3.332 after the removal of the outliers. Compared with a robust regression method, the result of the proposed method seemed more reasonable.

RODS: Rarity based Outlier Detection in a Sparse Coding Framework

Outlier detection has been an active area of research for a few decades. We propose a new definition of outlier that is useful for high-dimensional data. According to this definition, given a dictionary of atoms learned using the sparse coding objective, the outlierness of a data point depends jointly on two factors: the frequency of each atom in reconstructing all data points (or its negative log activity ratio, NLAR) and the strength by which it is used in reconstructing the current point. A R arity based O utlier D etection algorithm in a S parse coding framework (RODS) that consists of two components, NLAR learning and outlier scoring, is developed. This algorithm is unsupervised; both the offline and online variants are presented. It is governed by very few manually-tunable parameters and operates in linear time. We demonstrate the superior performance of the RODS in comparison with various state-of-the-art outlier detection algorithms on several benchmark datasets. We also demonstrate its effectiveness using three real-world case studies: saliency detection in images, abnormal event detection in videos, and change detection in data streams. Our evaluations shows that RODS outperforms competing algorithms reported in the outlier detection, saliency detection, video event detection, and change detection literature.

Toward Generalizing the Unification with Statistical Outliers

In this work, we introduce a novel definition of outlier, namely the Gradient Outlier Factor (or GOF), with the aim to provide a definition that unifies with the statistical one on some standard distributions but has a different behavior in the presence of mixture distributions. Intuitively, the GOF score measures the probability to stay in the neighborhood of a certain object. It is directly proportional to the density and inversely proportional to the variation of the density. We derive formal properties under which the GOF definition unifies the statistical outlier definition and show that the unification holds for some standard distributions, while the GOF is able to capture tails in the presence of different distributions even if their densities sensibly differ. Moreover, we provide a probabilistic interpretation of the GOF score, by means of the notion of density of the data density. Experimental results confirm that there are scenarios in which the novel definition can be profitably employed. To the best of our knowledge, except for distance-based outlier, no other data mining outlier definition has a so clearly established relationship with statistical outliers.

Index Based Hidden Outlier Detection in Metric Space

Useless and noise information occupies large amount of big data, which increases our difficulty to extract worthy information. Therefore outlier detection attracts much attention recently, but if two points are far from other points but are relatively close to each other, they are less likely to be detected as outliers because of their adjacency to each other. In this situation, outliers are hidden by each other. In this paper, we propose a new perspective of hidden outlier. Experimental results show that it is more accurate than existing distance-based definitions of outliers. Accordingly, we exploit a candidate set based hidden outlier detection (HOD) algorithm. HOD algorithm achieves higher accuracy with comparable running time. Further, we develop an index based HOD (iHOD) algorithm to get higher detection speed.