Types Of Outliers Research Articles

(1909-5457) A robust least squares fuzzy regression model based on kernel function

In this paper, a new approach is presented to fi t a robust fuzzy regression model based on some fuzzy quantities. In thisapproach, we fi rst introduce a new distance between two fuzzy numbers using the kernel function, and then, based onthe least squares method, the parameters of fuzzy regression model is estimated. The proposed approach has a suitableperformance to present the robust fuzzy model in the presence of different types of outliers. Using some simulated datasets and some real data sets, the application of the proposed approach in modeling some characteristics with outliers,is studied.

Analysis of Generalized Space Time Autoregressive with Exogenous Variable (GSTARX) Model with Outlier Factor

The outlier is an observation data that has different characteristics from others. Frequently, outliers are removed to improve the accuracy of the estimators. But sometimes the presence of an outlier has a specific meaning, which explanation can be lost if the outlier is removed. There are two exceptional cases from types of outliers, Innovative Outlier (IO) and Additive Outlier (AO). The presence of an outlier in the space-time model is no exception. Space-time model, not only influenced by previous observations at the same location and previous observations in a different location, or there are not only time and location dependencies, but also there are some other things that affect, which can be expressed as an exogenous variable. GSTARX is a model that combines not only time and location but also involves exogenous variables. In the GSTARX model, the presence of outliers may also be detected and may have spatial correlation at a time. In this paper, the iterative procedure in detecting outliers in the GSTARX model was introduced. Therefore data containing outliers is not deleted or ignored but still involves the outlier data by adding an outlier factor to the GSTARX model. The power of the procedure in detecting outliers is investigated by simulation experiments. The result is a GSTARX model with outlier factors that maintain the outlier factor.

Robust feature screening procedures for single and mixed types of data

ABSTRACTFeature screening procedures aim to reducing the dimensionality of data with exponentially-growing dimensions. Existing procedures all focused on a single type of predictors, which are either all continuous or all discrete. They cannot address mixed types of variables, outliers, or nonlinear trends. In this paper we first propose new feature screening procedure(s) for different continuous/discrete combinations of response and predictor variables. They are respectively based on marginal Spearman correlation, marginal ANOVA test, marginal Kruskal-Wallis test, Kolmogorov-Smirnov test, Mann-Whitney test, and smoothing splines modeling. Extensive simulation studies are performed to compare the new and existing procedures, with the aim of identifying a best robust screening procedure for each single type of data. Then we combine these best screening procedures to form the robust feature screening procedure for mixed type of data. We demonstrate its robustness against outliers and model misspecification through simulation studies and a real example.

Outlier Detection for Water Supply Data Based on Joint Auto-Encoder

With the development of science and technology, the status of the water environment has received more and more attention. In this paper, we propose a deep learning model, named a Joint Auto-Encoder network, to solve the problem of outlier detection in water supply data. The Joint Auto-Encoder network first expands the size of training data and extracts the useful features from the input data, and then reconstructs the input data effectively into an output. The outliers are detected based on the network’s reconstruction errors, with a larger reconstruction error indicating a higher rate to be an outlier. For water supply data, there are mainly two types of outliers: outliers with large values and those with values closed to zero. We set two separate thresholds, τ₁ and τ₂, for the reconstruction errors to detect the two types of outliers respectively. The data samples with reconstruction errors exceeding the thresholds are voted to be outliers. The two thresholds can be calculated by the classification confusion matrix and the receiver operating characteristic (ROC) curve. We have also performed comparisons between the Joint Auto-Encoder and the vanilla Auto-Encoder in this paper on both the synthesis data set and the MNIST data set. As a result, our model has proved to outperform the vanilla Auto-Encoder and some other outlier detection approaches with the recall rate of 98.94 percent in water supply data.

Review of Outlier Detection and Identifying Using Robust Regression Model

Outliers are observations that have extreme value relations. Herewith leverage is a measure of how an independent variable deviates from its mean. An observation with an extreme value on a predictor variable is a point with high leverage. The presence of outliers can lead to inflated error rates and substantial distortions of parameter and statistic estimates when using either parametric or nonparametric tests. Casual observation of the literature suggests that researchers rarely report checking for outliers of any sort and taking remedial measures for outliers. Outliers can have positive deleterious effects on statistical analyses. For instance, they serve to increase error variance and reduce the power of statistical tests; they can decrease normality, altering the odds of making both Type I and Type II errors for non- randomly distributed; and they can seriously bias or influence estimates that may be of substantive interest. These outliers are cased from incorrect recording data, intentional or motivated mis-reporting, sampling error and Outliers as legitimate cases sampled from the correct population. According to some literatures; Point outliers, Contextual Outliers and Collective Outliers are the three types of outliers. Robust regression estimators can be a powerful tool for detection and identifying outliers in complicated data sets. Robust regression, deals with the problem of outliers in a regression and produce different coefficient estimates than OLS does.

On the Analysis and Modelling of the Harmonized Consumer Price Indices of West African Economic and Monetary Union Member States

The major goal of this paper is a better understanding of the price dynamics of the eight West African Economic and Monetary Union (WAEMU) member states. More specifically, the study intends to find the best models with suitable forecasting power for the monthly Harmonized Consumer Price Indices (HCPI) of each of the WAEMU countries. Descriptive statistics and time series modeling approaches were applied to the HCPI base 100=2008 series covering the period from January 1998 to December 2019. The analysis revealed that Guinea-Bissau had the highest average HCPI of 99.88 and Senegal the lowest of 93.73. Togo attained the highest HCPI of 119.60 and Benin the lowest of 71.54 over the period studied. The indices of Togo and Guinea-Bissau have the highest and the smallest variance of 225.56 and 79.60, respectively. All the indices have an upward trend and contain cyclical and seasonal components. Using the Box-Jenkins methodology and Expert Modeler of SPSS five types of outliers, i.e. additive, additive patch, transient, innovational and level change, have been detected and different SARIMA models were proposed. Bartlett's B-test detects significant periodic effects in the residuals of the models for Burkina-Faso and Cote d’Ivoire. The residuals of all the models have been declared Gaussian by Shapiro-Wilks and Jarque-Bera normality tests while those of Cote d’Ivoire fail the latest test for normality due to the discrepancy of their skewness with that of a normal distribution. Adequacy of the claimed models has been corroborated by adequate values of key fitting and predicting statistics and the non-significance of the paired t-test on the mean difference between the observed and the adjusted values. Thus SARIMA (0,1,0) (0,1,1)12 model was found to best fit the HCPI for Burkina Faso, Cote d'Ivoire, Niger, Senegal and Togo; and the data for Benin, Guinea Bissau and Mali are found to be SARIMA (3,1,0) (1,0,1)12, SARIMA (0,1,0) (1,0,1)12 and SARIMA (1,1,1) (0,1,1)12 process, respectively. The differences between the retained models raise doubts on the claimed objective of convergence of the economies of the WAEMU countries. Engle's Lagrange Multiplier test for autoregressive conditional heteroscedasticity (ARCH) reveals the homoscedasticity of the residuals of all the models but the one of Cote d'Ivoire. Thus, for better modeling of the index of Cote d’Ivoire, a GARCH model may be envisioned.

INTRA-URBAN HEAT ISLAND DETECTION AND TREND CHARACTERIZATION IN METRO MANILA USING SURFACE TEMPERATURES DERIVED FROM MULTI-TEMPORAL LANDSAT DATA

Abstract. Unprecedented urbanization in Metro Manila has led to the proliferation of the urban heat island (UHI) effect. This is characterized by a prominent difference in the temperatures of the urban and its surrounding rural and less urbanized areas. Temperature differences occur within these UHI’s indicating the existence of intra-urban heat islands (IUHI). UHI’s and IUHI’s are well-documented indicators of urban environmental degradation and therefore puts the population of Metro Manila at risk. In anticipation of these effects, their detection and the characterization of their behaviour through time can contribute to proper urban planning thus mitigating harmful effects. Google Earth Engine was used to retrieve land surface temperatures (LST) from Landsat data from 1997 to 2019 using emissivity estimation. The Local Moran’s I statistic was then used to identify cluster and outlier types (COT). A histogram with 10 bins representing the net COT frequencies per barangay was then used to identify IUHI’s. Annual temperature measurements and COT areas were plotted against time and based on linear-fit trend lines they characterize the study area as to having an annual increase in temperature of roughly 0.18 °C and hotspot area extent of around 0.03 km2, and a decrease in coldspot area extent around 0.01 km2. Hotspots were found to be frequent in the cities of Caloocan, Manila, Pasay, and Quezon while coldspots were found to be frequent in the cities of Caloocan, Las Piñas, Malabon, Navotas, and Valenzuela. In conclusion, IUHI’s were detected with statistical basis, both spatially and temporally.

Comparisons of outlier tests for potency bioassays.

Potency bioassays are used to measure biological activity. Consequently, potency is considered a critical quality attribute in manufacturing. Relative potency is measured by comparing the concentration-response curves of a manufactured test batch with that of a reference standard. If the curve shapes are deemed similar, the test batch is said to exhibit constant relative potency with the reference standard, a critical requirement for calibrating the potency of the final drug product. Outliers in bioassay potency data may result in the false acceptance/rejection of a bad/good sample and, if accepted, may yield a biased relative potency estimate. To avoid these issues, the USP<1032> recommends the screening of bioassay data for outliers prior to performing a relative potency analysis. In a recently published work, the effects of one or more outliers, outlier size, and outlier type on similarity testing and estimation of relative potency were thoroughly examined, confirming the USP<1032> outlier guidance. As a follow-up, several outlier detection methods, including those proposed by the USP<1010>, are evaluated and compared in this work through computer simulation. Two novel outlier detection methods are also proposed. The effects of outlier removal on similarity testing and estimation of relative potency were evaluated, resulting in recommendations for best practice.

Resolving Water, Proteins, and Lipids from In Vivo Confocal Raman Spectra of Stratum Corneum through a Chemometric Approach.

Development of this in vivo confocal Raman spectroscopic method enables the direct measurement of water, proteins, and lipids with depth resolution in human subjects. This information is very important for skin-related diseases and characterizing skin care product performance. This protocol illustrates a method for confocal Raman spectra collection and the subsequent analysis of the spectral dataset leveraging chemometrics. The goal of this method is to establish a standard protocol for data collection and provide general guidance for data analysis. Preprocessing (e.g., removal of outlier spectra) is a critical step when processing large datasets from clinical studies. As an example, we provide guidance based on prior knowledge of a dataset to identify the types of outliers and develop specific strategies to remove them. A principal component analysis is performed, and the loading spectra are compared with spectra from reference materials to select the number of components used in the final multivariate curve resolution (MCR) analysis. This approach is successful for extracting meaningful information from a large spectral dataset.

Parallel graph-based anomaly detection technique for sequential data

In data mining, outlier detection is applied in different domains. It has very large applications such as energy consumption analysis, forecasting hurricanes in meteorological data, fraud and intrusion detection, event detection and system monitoring in sensor networks, etc. Most of existing outlier detection techniques depend on the properties of a particular type of data and can not deal with a large volume of data well, which mean that there is a necessity for improved methodologies and techniques to be applied to a large amount of data with different types in other application areas. In this paper, a parallel outlier detection technique is developed to detect the outliers in the sequential data. Although there are many types of outliers, this paper concentrates on the contextual anomalies. The proposed technique uses a graph approach to detect the outliers. It is very flexible, fast, and no labeled data is needed comparing to many previous approaches. The experimental results show the detected contextual outliers in the sequential data, as well as the efficient scaling up to handle the massive data by increasing the number of processors. The results prove that the parallelism of the proposed technique is very valuable.

A new fast filtering algorithm for a 3D point cloud based on RGB-D information.

A point cloud that is obtained by an RGB-D camera will inevitably be affected by outliers that do not belong to the surface of the object, which is due to the different viewing angles, light intensities, and reflective characteristics of the object surface and the limitations of the sensors. An effective and fast outlier removal method based on RGB-D information is proposed in this paper. This method aligns the color image to the depth image, and the color mapping image is converted to an HSV image. Then, the optimal segmentation threshold of the V image that is calculated by using the Otsu algorithm is applied to segment the color mapping image into a binary image, which is used to extract the valid point cloud from the original point cloud with outliers. The robustness of the proposed method to the noise types, light intensity and contrast is evaluated by using several experiments; additionally, the method is compared with other filtering methods and applied to independently developed foot scanning equipment. The experimental results show that the proposed method can remove all type of outliers quickly and effectively.

Outlier Aware Network Embedding for Attributed Networks

Attributed network embedding has received much interest from the research community as most of the networks come with some content in each node, which is also known as node attributes. Existing attributed network approaches work well when the network is consistent in structure and attributes, and nodes behave as expected. But real world networks often have anomalous nodes. Typically these outliers, being relatively unexplainable, affect the embeddings of other nodes in the network. Thus all the downstream network mining tasks fail miserably in the presence of such outliers. Hence an integrated approach to detect anomalies and reduce their overall effect on the network embedding is required.Towards this end, we propose an unsupervised outlier aware network embedding algorithm (ONE) for attributed networks, which minimizes the effect of the outlier nodes, and hence generates robust network embeddings. We align and jointly optimize the loss functions coming from structure and attributes of the network. To the best of our knowledge, this is the first generic network embedding approach which incorporates the effect of outliers for an attributed network without any supervision. We experimented on publicly available real networks and manually planted different types of outliers to check the performance of the proposed algorithm. Results demonstrate the superiority of our approach to detect the network outliers compared to the state-of-the-art approaches. We also consider different downstream machine learning applications on networks to show the efficiency of ONE as a generic network embedding technique. The source code is made available at https://github.com/sambaranban/ONE.

Deteksi Pencilan pada Model ARIMA dengan Bayesian Information Criterion (BIC) Termodifikasi

Time series data may be affected by special events or circumstances such as promotions, natural disasters, etc. These events can lead to inconsistent observations in the series called outliers. Because outliers can make invalid conclusions, it is important to carry out procedures in detecting outlier effects. In outlier detection there is one type of outlier, namely additive outlier (AO). The process of detecting additive outliers in the ARIMA model can be said as a model selection problem, where the candidate model assumes additive outliers at a certain time. In the selection of models there are criteria that must be considered in order to produce the best model. The good criteria for models selection can use the Bayesian Information Criterion (BIC) derived by Schwarz (1978). Galeano and Pena (2011) proposed a modified Bayesian Information Criterion for model selection and detect potential outliers. The modified Bayesian Information Criterion for outlier detection will be applied to the data OutStanding Loan PT.Pegadaian Cimahi year 2013-2017. So that the best model is obtained that the model with adding 2 potential outliers with the ARIMA model (1.0,0), that outliers at observations 48, and 58 because it has a minimum BICUP value of 1064.95650.

A Decomposition of Total Variation Depth for Understanding Functional Outliers

There has been extensive work on data depth-based methods for robust multivariate data analysis. Recent developments have moved to infinite-dimensional objects, such as functional data. In this work, we propose a notion of depth, the total variation depth, for functional data, which has many desirable features and is well suited for outlier detection. The proposed depth is in the form of an integral of a univariate depth function. We show that the novel formation of the total variation depth leads to useful decomposition associated with shape and magnitude outlyingness of functional data. Compared to magnitude outliers, shape outliers are often masked among the rest of samples and more difficult to identify. We then further develop an effective procedure and visualization tools for detecting both types of outliers, while naturally accounting for the correlation in functional data. The outlier detection performance is investigated through simulations under various outlier models. Finally, the proposed methodology is demonstrated using real datasets of curves, images, and video frames.

Errors in Time-Series Remote Sensing and an Open Access Application for Detecting and Visualizing Spatial Data Outliers Using Google Earth Engine

Remotely sensed measures of productivity are frequently used to characterize global agriculture and vegetated ecosystems, and are often downscaled to describe local, remote areas where finer spatial and temporal resolution data are regularly unavailable. While data errors may propagate throughout any analytical procedure, those that are missed during delivery and preliminary data mining require more attention. Here, a collection of formerly and presently available global remote sensing products are compiled to demonstrate the temporal and geographic breadth of remote sensing uncertainty. Vegetation productivity measures are invaluable for monitoring global health, but erroneous estimates that go unrecognized may result in serious policy mistakes. It is eminently clear that generalizable and accessible a priori methods for anomaly detection are lacking and urgently needed so that data errors are recognized before public delivery and before widespread use. Simple yet effective statistics such as the modified Z-score, Tukey's outliers, and Geary's C are leveraged here to identify, locate, and visualize the types of outliers that remote sensing data users may elect to omit or correct. Contributing to the growing ensemble of Google Earth Engine methodologies, we propose this generalizable method of detecting spatial outliers for remote sensing error management by users across scientific domains.

A robust sparse linear approach for contaminated data

A challenging problem in a linear regression model is to select a parsimonious model which is robust to the presence of contamination in the data. In this paper, we present a sparse linear approach which detects outliers by using a highly robust regression method. The model with optimal predictive ability as measured by the median absolute deviation of the prediction errors on JackKnife subsets is used to detect outliers. The performance of the proposed method is evaluated on a simulation study with a different type of outliers and high leverage points and also on a real data set.

Robust curve registration using the t distribution

In functional data analysis, it is often of interest to discover a general common pattern, or shape, of the function. When the subject-specific amplitude and phase variation of data are not of interest, curve registration can be used to separate the variation from the data. Shape-invariant models (SIM), one of the registration methods, aim to estimate the unknown shared-shape function. However, the use of SIM and of general registration methods assumes that all curves have the shared-shape in common and does not consider the existence of outliers, such as a curve, whose shape is inconsistent with the remainder of the data. Therefore, we propose using the t distribution to robustify SIMs, allowing outliers of amplitude, phase, and other errors. Our SIM can identify and classify the three types of outliers mentioned above. We use simulation and an empirical data set to evaluate the performance of our robust SIM.

A robust functional time series forecasting method

ABSTRACTUnivariate time series often take the form of a collection of curves observed sequentially over time. Examples of these include hourly ground-level ozone concentration curves. These curves can be viewed as a time series of functions observed at equally spaced intervals over a dense grid. Since functional time series may contain various types of outliers, we introduce a robust functional time series forecasting method to down-weigh the influence of outliers in forecasting. Through a robust principal component analysis based on projection pursuit, a time series of functions can be decomposed into a set of robust dynamic functional principal components and their associated scores. Conditioning on the estimated functional principal components, the crux of the curve-forecasting problem lies in modelling and forecasting principal component scores, through a robust vector autoregressive forecasting method. Via a simulation study and an empirical study on forecasting ground-level ozone concentration, the robust method demonstrates the superior forecast accuracy that dynamic functional principal component regression entails. The robust method also shows the superior estimation accuracy of the parameters in the vector autoregressive models for modelling and forecasting principal component scores, and thus improves curve forecast accuracy.

Distinguishing protest responses in contingent valuation: A conceptualization of motivations and attitudes behind them.

The percentage of protesters in contingent valuation surveys is substantial–about 20% across many studies. This paper seeks to clarify the motivations behind protest responses. In addition, the question whether the estimation of willingness to pay (WTP) is more biased by the exclusion or inclusion of protest bids is yet undecided. Methodological improvements are difficult for three reasons: motivations behind protest responses are largely unclear, definitions of protest differ between studies and often only participants who state a zero WTP are asked for their reasons. Our survey on farm animal welfare (n = 1335) provides detailed motivations, two definitions and includes debriefing of all participants for their WTP. We find that protest bids are not a refusal to answer, they are neither irrational nor driven by lack of understanding. Quite the contrary, a large part of participants is directly motivated by moral reasons. Furthermore, protest responses are not coupled to a zero WTP. In our sample, only 8% out of 32% protesting participants had a zero WTP. Only a small fraction of zero bids (0.4%) are true WTP-statements, i.e. respondents were satisfied with the status quo. This finding has important implications for existing WTP-estimates which might be biased. Finally, we provide detailed estimates of the WTP for animal welfare issues by including and excluding different types of protesters and outliers.

Integration of deep feature extraction and ensemble learning for outlier detection

It is obvious to see that most of the datasets do not have exactly equal number of samples for each class. However, there are some tasks like detection of fraudulent transactions, for which class imbalance is overwhelming and one of the classes has very low (even less than 10% of the entire data) amount of samples. These tasks often fall under outlier detection. Moreover, there are some scenarios where there may be multiple subsets of the outlier class. In such cases, it should be treated as a multiple outlier type detection scenario. In this article, we have proposed a system that can efficiently handle all the aforementioned problems. We have used stacked autoencoders to extract features and then used an ensemble of probabilistic neural networks to do a majority voting and detect the outliers. Such a system is seen to have a better and reliable performance as compared to the other outlier detection systems in most of the datasets tested upon. It is seen that use of autoencoders clearly enhanced the outlier detection performance.