Outlying Data Points Research Articles

Application of statistical geo-spatial information technology to soil stratification in the Seoul metropolitan area

Typical geotechnical testing results reflect the level of soil uncertainty, which requires statistical corrections of the data for an appropriate engineering decision. This study proposes frameworks to detect outlying data points using statistical analyses, the cross-validation-based method and the generalised extreme value distribution-based method. The borehole data regarding soil depth distribution in a central area of Seoul, South Korea are assessed to validate the aforementioned methods for comparison with the distribution-based method and the Moran scatterplot method. The results show that the proposed methods enable more reliable spatial distributions to be achieved with a quantitative evaluation of local reliability.

Fuzzy least-absolutes regression using shape preserving operations

The purpose of this study is to introduce a new fuzzy regression model, based on the least-absolutes method. The fuzzy simple and fuzzy multivariate regression models with fuzzy input-fuzzy output are considered in which the coefficients of the models are themselves fuzzy. The proposed method is based on a new metric on the space of LR fuzzy numbers. Fuzzy arithmetic operations are based on the weakest triangular norm, TW, which is the unique triangular norm that preserves the shape of fuzzy numbers during multiplication. The results of comparative studies and numerical examples indicate that, using the similarity measure criterion as well as a predictive ability index, the proposed method has a better performance than the least-squares method, especially when the data set includes some outlier data point(s).

CAG repeat expansion in Huntington disease determines age at onset in a fully dominant fashion

Age at onset of diagnostic motor manifestations in Huntington disease (HD) is strongly correlated with an expanded CAG trinucleotide repeat. The length of the normal CAG repeat allele has been reported also to influence age at onset, in interaction with the expanded allele. Due to profound implications for disease mechanism and modification, we tested whether the normal allele, interaction between the expanded and normal alleles, or presence of a second expanded allele affects age at onset of HD motor signs. We modeled natural log-transformed age at onset as a function of CAG repeat lengths of expanded and normal alleles and their interaction by linear regression. An apparently significant effect of interaction on age at motor onset among 4,068 subjects was dependent on a single outlier data point. A rigorous statistical analysis with a well-behaved dataset that conformed to the fundamental assumptions of linear regression (e.g., constant variance and normally distributed error) revealed significance only for the expanded CAG repeat, with no effect of the normal CAG repeat. Ten subjects with 2 expanded alleles showed an age at motor onset consistent with the length of the larger expanded allele. Normal allele CAG length, interaction between expanded and normal alleles, and presence of a second expanded allele do not influence age at onset of motor manifestations, indicating that the rate of HD pathogenesis leading to motor diagnosis is determined by a completely dominant action of the longest expanded allele and as yet unidentified genetic or environmental factors.

Erratum: Thermalization and its mechanism for generic isolated quantum systems

Nature 452, 854–858 (2008). In Figs 3b and e of this Letter, there were two points that could have been misinterpreted as outlying data points (they were inadvertently printed parts of a key to the figure that was incompletely removed). These points have been removed in the HTML and PDF versions. Wethank David S.

Improved Time-Series Photometry and Calibration Method for Non-Crowded Fields: MMT Megacam and HAT-South Experiences

AbstractWe present a new photometric reduction method for precise time-series photometry of non-crowded fields that does not need to involve relatively complicated and CPU intensive techniques such as point-spread-function (PSF) fitting or difference image analysis. This method, which combines multi-aperture index photometry and a spatio-temporal de-trending algorithm, gives much superior performance in data recovery and light-curve precision. In practice, the brutal filtering that is often applied to remove outlying data points can result in the loss of vital data, with seriously negative impacts on short-term variations such as flares. Our method utilizes nearly 100% of available data and reduces the rms scatter to several times smaller than that for archived light curves for brighter stars. We outline the details of our new method, and apply it to cases of sample data from the MMT survey of the M37 field, and the HAT-South survey.

Comprehensive Utilization of Temporal and Spatial Domain Outlier Detection Methods for Mobile Terrestrial LiDAR Data

Terrestrial LiDAR provides many disciplines with an effective and efficient means of producing realistic three-dimensional models of real world objects. With the advent of mobile terrestrial LiDAR, this ability has been expanded to include the rapid collection of three-dimensional models of large urban scenes. For all its usefulness, it does have drawbacks. One of the major problems faced by the LiDAR industry today is the automatic removal of outlying data points from LiDAR point clouds. This paper discusses the development and combined implementation of two methods of performing outlier detection in georeferenced point clouds. These methods made use of the raw data available from most time-of-flight mobile terrestrial LiDAR scanners in both the temporal and spatial domains. The first method involved a moving fixed interval smoother derived from the well-known position velocity acceleration Kalman Filter. The second method fitted a quadratic curved surface to sections of LiDAR data. The combined use of these routines is discussed through examples with real LiDAR data.

ANALYSIS OF STRUCTURAL PLASTICITY IN THE HONEY BEE BRAIN USING THE CAVALIERI ESTIMATOR OF VOLUME AND THE DISECTOR METHOD

The antennal lobe of the worker honey bee has been used as a model system to address the origins of structural plasticity in the central nervous system. A combination of light and electron microscopy was used to determine total synapse number within an easily identifiable sub-unit of the antennal lobe neuropil, the T4-2(1) glomerulus. The Cavalieri method was applied at the light microscope level to determine a reference volume (Vref) of this glomerulus. Using transmission electron microscopy, the physical disector was used to determine synaptic density (Nv) within the T4-2(1) glomerulus. An estimate of the total synapse number N(syn) was determined by; N(syn) = V(ref) Nv. Newly emerged, 4-day old,10-day old and forager-aged bees were analysed. Results showed that despite a significant increase in T4-2(1) volume with age, the total number of synapses in this glomerulus did not show a corresponding increase. Disturbingly, it is possible that huge variances within age groups, due to one or two outlying data points, could be masking the true trend of synapse number. This variance, the heterogeneous distribution of synapses within this glomerulus and the problems associated with reproducibility of synapse counts are discussed.

The flood algorithm—a multivariate, self-organizing-map-based, robust location and covariance estimator

Self-organizing maps (SOMs) introduced by Kohonen (Biol. Cybern. 43(1):59---69, 1982) are well-known in the field of artificial neural networks. The way SOMs are performing is very intuitive, leading to great popularity and numerous applications (related to statistics: classification, clustering). The result of the unsupervised learning process performed by SOMs is a non-linear, low-dimensional projection of the high-dimensional input data, that preserves certain features of the underlying data, e.g.?the topology and probability distribution (Lee and Verleysen in Nonlinear Dimensionality Reduction, Springer, 2007; Kohonen in Self-organizing Maps, 3rd edn., Springer, 2001). With the U-matrix Ultsch (Information and Classification: Concepts, Methods and Applications, pp. 307---313, Springer, 1993) introduced a powerful visual representation of the SOM results. We propose an approach that utilizes the U-matrix to identify outlying data points. Then the revised subsample (i.e. the initial sample minus the outlying points) is used to give a robust estimation of location and scatter.

Developing an accident modification function for speed enforcement

This paper presents an accident modification function for speed enforcement. The function describes the percentage effect of enforcement on the number of injury accidents as a function of the relative change in the amount of enforcement. Thirteen studies that have evaluated the effects of different levels of enforcement, each containing between four and eight data points, were used as data source. A hypothesis regarding the shape of an accident modification function was formulated to help assess the plausibility of the estimated relationships. In the first stage of analysis, functions were fitted to the data points of each study by means of weighted regression analysis. Many of these functions were highly implausible and likely to be influenced by outlying data points and random variation. In the second stage of analysis, data from all studies except two were pooled and data points referring to identical or almost identical levels of enforcement were aggregated to reduce the influence of random variation. The number of data points was thus reduced from 63 to 15 and an accident modification function was fitted to the data points. The function was consistent with the hypothesis guiding the study.

Procedures to improve the repeatability of forced oscillation measurements in school-aged children

Forced oscillation technique (FOT) parameters are less repeatable than spirometry, and the impact of technical factors, such as data acquisition and data filtering, are unknown. FOT was performed, in triplicate, on 48 children (8-11 years) and repeated two weeks later. We examined the separate effects of monitoring tidal volume (V(T)) prior to measurement and length of data acquisition on measurement repeatability. We compared the effects on repeatability of a filtering technique in which complete breaths containing respiratory artefact were rejected and statistical filters in which outlying data points were rejected. Within- and between-session repeatability of respiratory system resistance (Rrs) and reactance (Xrs) were assessed using coefficient of variation (CV) and intra-class correlation coefficient (ICC). Longer data acquisition reduced CV of Rrs and Xrs (60s vs. shorter durations, p ≤ 0.001). Monitoring V(T) reduced CV of Rrs (p = 0.05). Complete breath filtering improved CV and ICC for both Rrs and Xrs. The repeatability of FOT measurements can be improved by optimising data acquisition and filtering.

Robustness and inference in nonparametric partial frontier modeling

A major aim in recent nonparametric frontier modeling is to estimate a partial frontier well inside the sample of production units but near the optimal boundary. Two concepts of partial boundaries of the production set have been proposed: an expected maximum output frontier of order m = 1 , 2 , … and a conditional quantile-type frontier of order α ∈ ] 0 , 1 ] . In this paper, we answer the important question of how the two families are linked. For each m , we specify the order α for which both partial production frontiers can be compared. We show that even one perturbation in data is sufficient for breakdown of the nonparametric order- m frontiers, whereas the global robustness of the order- α frontiers attains a higher breakdown value. Nevertheless, once the α frontiers break down, they become less resistant to outliers than the order- m frontiers. Moreover, the m frontiers have the advantage to be statistically more efficient. Based on these findings, we suggest a methodology for identifying outlying data points. We establish some asymptotic results, contributing to important gaps in the literature. The theoretical findings are illustrated via simulations and real data.

Non-parametric linear regression of discrete Fourier transform convoluted chromatographic peak responses under non-ideal conditions of internal standard method

This manuscript discusses the application of chemometrics to the handling of HPLC response data using the internal standard method (ISM). This was performed on a model mixture containing terbutaline sulphate, guaiphenesin, bromhexine HCl, sodium benzoate and propylparaben as an internal standard. Derivative treatment of chromatographic response data of analyte and internal standard was followed by convolution of the resulting derivative curves using 8-points sin x i polynomials (discrete Fourier functions). The response of each analyte signal, its corresponding derivative and convoluted derivative data were divided by that of the internal standard to obtain the corresponding ratio data. This was found beneficial in eliminating different types of interferences. It was successfully applied to handle some of the most common chromatographic problems and non-ideal conditions, namely: overlapping chromatographic peaks and very low analyte concentrations. For example, a significant change in the correlation coefficient of sodium benzoate, in case of overlapping peaks, went from 0.9975 to 0.9998 on applying normal conventional peak area and first derivative under Fourier functions methods, respectively. Also a significant improvement in the precision and accuracy for the determination of synthetic mixtures and dosage forms in non-ideal cases was achieved. For example, in the case of overlapping peaks guaiphenesin mean recovery% and RSD% went from 91.57, 9.83 to 100.04, 0.78 on applying normal conventional peak area and first derivative under Fourier functions methods, respectively. This work also compares the application of Theil's method, a non-parametric regression method, in handling the response ratio data, with the least squares parametric regression method, which is considered the de facto standard method used for regression. Theil's method was found to be superior to the method of least squares as it assumes that errors could occur in both x- and y-directions and they might not be normally distributed. In addition, it could effectively circumvent any outlier data points. For the purpose of comparison, the results obtained using the above described internal standard method were compared with the external standard method for all types of linearity.

Effects of physiological noise in population analysis of diffusion tensor MRI data

The goal of this study is to characterize the potential effect of artifacts originating from physiological noise on statistical analysis of diffusion tensor MRI (DTI) data in a population. DTI derived quantities including mean diffusivity (Trace(D)), fractional anisotropy (FA), and principal eigenvector (ε1) are computed in the brain of 40 healthy subjects from tensors estimated using two different methods: conventional nonlinear least-squares, and robust fitting (RESTORE). RESTORE identifies artifactual data points as outliers and excludes them on a voxel-by-voxel basis. We found that outlier data points are localized in specific spatial clusters in the population, indicating a consistency in brain regions affected across subjects. In brain parenchyma RESTORE slightly reduces inter-subject variance of FA and Trace(D). The dominant effect of artifacts, however, is bias. Voxel-wise analysis indicates that inclusion of outlier data points results in clusters of under- and over-estimation of FA, while Trace(D) is always over-estimated. Removing outliers affects ε1 mostly in low anisotropy regions. It was found that brain regions known to be affected by cardiac pulsation – cerebellum and genu of the corpus callosum, as well as regions not previously reported, splenium of the corpus callosum–show significant effects in the population analysis. It is generally assumed that statistical properties of DTI data are homogenous across the brain. This assumption does not appear to be valid based on these results. The use of RESTORE can lead to a more accurate evaluation of a population, and help reduce spurious findings that may occur due to artifacts in DTI data.

Similarity-Dissimilarity Plot for Visualization of High Dimensional Data in Biomedical Pattern Classification

In pattern classification problems, feature extraction is an important step. Quality of features in discriminating different classes plays an important role in pattern classification problems. In real life, pattern classification may require high dimensional feature space and it is impossible to visualize the feature space if the dimension of feature space is greater than four. In this paper, we have proposed a Similarity-Dissimilarity plot which can project high dimensional space to a two dimensional space while retaining important characteristics required to assess the discrimination quality of the features. Similarity-dissimilarity plot can reveal information about the amount of overlap of features of different classes. Separable data points of different classes will also be visible on the plot which can be classified correctly using appropriate classifier. Hence, approximate classification accuracy can be predicted. Moreover, it is possible to know about whom class the misclassified data points will be confused by the classifier. Outlier data points can also be located on the similarity-dissimilarity plot. Various examples of synthetic data are used to highlight important characteristics of the proposed plot. Some real life examples from biomedical data are also used for the analysis. The proposed plot is independent of number of dimensions of the feature space.

Developing nonlinear queuing regressions to increase emergency department patient safety: Approximating reneging with balking

Administrators know when Emergency Department (ED) overcrowding is a problem in their hospital. Lead times to change ED capacity are long and require strategic tools. ED patients who Leave WithOut Treatment (LWOT) before seeing a physician are, in queuing nomenclature, ‘reneging’ from an overcrowded situation and are an important measure of ED patient safety. We propose to enable strategic decision making on future ED capacity on the basis of patient safety (rather than congestion measures). We hypothesize that the LWOT reneging percentage is captured by the balking probability ( p K ) relationship of an M/M/1/K queue. If true, this relationship is superior to the typical ad hoc regression relationships commonly found. Since it is based on a physical scientific mechanism, the sample size requirements and extrapolation power are improved. We derive the form of a binomial response nonlinear weighted regression model that best fits p K for predicting LWOT to long-term ED performance by means of Gauss–Newton linearization. Our results include asymptotic Wald confidence intervals on prediction, specific Pearson and Deviance model goodness-of-fit tests, and residual analysis that facilitate identification of outlying data points. None of these features exist for reneging (or balking) models previously presented in the literature.

Empirical approaches for fast robust inversion of seismic moment tensor from surface waves

We present a method of robust inversion (also called IRLS: Iterative Reweighted Least Squares) which is surprisingly insensitive to outlier data points, as it discards automatically the aberrant points, without the necessity for careful human inspection and control of data quality. Different criteria based on residuals and signal‐to‐noise ratio are injected into the covariance matrix (acting like a weighting function), to perform the robust inversion using the iterative generalized discrete inverse method of Tarantola‐Valette. From a practical point of view, this algorithm is used as the Preliminary Determination of Focal Mechanism (PDFM) method, which is a project for rapid estimation of source parameters of strong earthquakes in the context of tsunami warning. The input data to be inverted are spectra of long period surface waves, and as an output, the computed result is the seismic moment tensor, from which focal geometry of an earthquake, and principal stress axes are obtained. This method can be applied to any other method of non‐linear (iterative) inversions confronted to the problem of outlier points polluting the data sets.

The analysis of outlying data points by robust Locally Weighted Scatter Plot Smooth: a model for the identification of problem banks

Continuing bank failures point to the need for early warning problem bank identification models to guide the actions of regulators and investors. Several models have been shown to work well, but most require extensive data preparation/manipulation and custom computer programmes to analyse the data, which hinders widespread adoption. In this article, we show that robust Locally Weighted Scatter Plot Smooth, a type of Local Regression Smoothing, which requires minimal data preparation and can be run in many off the shelf statistical packages such as SAS and SPSS, can be just as effective as an early warning system.

Modeling of variable speed refrigerated display cabinets based on adaptive support vector machine

In this paper the adaptive support vector machine (ASVM) method is introduced to the field of intelligent modeling of refrigerated display cabinets and used to construct a highly precise mathematical model of their performance. A model for a variable speed open vertical display cabinet was constructed using preprocessing techniques for measured data, including the elimination of outlying data points by the use of an exponential weighted moving average (EWMA). Using dynamic loss coefficient adjustment, the adaptation of the SVM for use in this application was achieved. From there, the object function for energy use per unit of display area total energy consumption (TEC)/total display area (TDA) was constructed and solved using the ASVM method. When compared to the results achieved using a back-propagation neural network (BPNN) model, the ASVM model for the refrigerated display cabinet was characterized by its simple structure, fast convergence speed and high prediction accuracy. The ASVM model also has better noise rejection properties than that of original SVM model. It was revealed by the theoretical analysis and experimental results presented in this paper that it is feasible to model of the display cabinet built using the ASVM method.

A robust partial least squares regression method with applications

AbstractPartial least squares (PLS) regression is a linear regression technique developed to relate many regressors to one or several response variables. Robust methods are introduced to reduce or remove the effect of outlying data points. In this paper, we show that if the sample covariance matrix is properly robustified further robustification of the linear regression steps of the PLS algorithm becomes unnecessary. The robust estimate of the covariance matrix is computed by searching for outliers in univariate projections of the data on a combination of random directions (Stahel—Donoho) and specific directions obtained by maximizing and minimizing the kurtosis coefficient of the projected data, as proposed by Peña and Prieto [1]. It is shown that this procedure is fast to apply and provides better results than other methods proposed in the literature. Its performance is illustrated by Monte Carlo and by an example, where the algorithm is able to show features of the data which were undetected by previous methods. Copyright © 2008 John Wiley & Sons, Ltd.

Robust Probabilistic Multivariate Calibration Model

In this article we propose a robust probabilistic multivariate calibration (RPMC) model in an attempt to identify linear relationships between two sets of observed variables contaminated with outliers. Instead of the Gaussian assumptions that predominate in classical statistical models, RPMC is closely related with the multivariate Student t-distribution over noises and latent variables. Thus RPMC diminishes the effect of outlying data points by regulating the thickness of the distribution tails. RPMC is essentially a robustified version of the supervised probabilistic principal component analysis (SPPCA) that has emerged recently. We show that RPMC encompasses probabilistic principal component analysis and SPPCA as limiting cases. We also derive an efficient EM algorithm for parameter estimation in RPMC. Based on a probabilistic description of latent variables, we present a procedure for the detection of outliers. The experimental results from both simulated examples and real life data sets demonstrate the effectiveness and robustness of our proposed approach.