Shift Outlier Research Articles

Panel data modeling: Identifying and handling outliers with the VSOM approach

While modeling panel data, identifying outliers is crucial, yet visually spotting them can be challenging. However, not many methods exist for outlier detection in panel data models other than robust estimation methods. This study uses the variance shift outlier model (VSOM) approach to identify and handle outliers in panel data modeling, both in single and simultaneous equation modeling. This study applies the VSOM approach to GDP and FDI data from ten countries that are part of the ASEANChina Free Trade Area (ACFTA). The results demonstrate that the VSOM approach effectively handle outliers in panel data modeling, in both single and simultaneous equation model. Specifically, the VSOM model shows a lower sum of squared residuals (SSR) compared to the null model. This indicates that VSOM successfully reduces the impact of outliers, leading to improved model fit and accuracy in capturing the underlying relationships within the data. Some of the highlights of the proposed approach are:•Identifying the square of the standardized residual.•Generate the distribution of the squared standardized residuals by parametric bootstrapping.•Reducing the impact of observations identified as outliers by downweighting their variance through the addition of a D matrix.

Anomaly detection for geological carbon sequestration monitoring

Geological carbon sequestration (GCS) is a method to reduce the emissions of CO2 into the atmosphere. During GCS operations CO2 is captured from the atmosphere or industrial activities and stored in geological formations for permanent storage. Monitoring is an important element of GCS because it ensures that the stored CO2 remains safely contained in the intended formation during the long term. Additionally, monitoring wells can help to detect CO2 leaks, prompt remediation actions, and provide valuable information to optimize storage by monitoring the behavior of the CO2 over time. In this work, we propose a method for GCS anomaly detection based on an LSTM Autoencoder Neural Network and Isolation Forest. The LSTM-Autoencoder uses the monitor Bottomhole Pressure (BHP) response while CO2 is being injected into a geological structure. To account for the subsurface uncertainty, multiple subsurface model realizations are created, and using reservoir simulation, the multiple monitor BHP are generated to capture the subsurface uncertainty. Anomaly BHP points are detected using the residuals of the LSTM-Autoencoder and Isolation Forest. Additionally, an anomaly score based on the subsurface uncertainty is proposed. Finally, the method robustness is evaluated using point outliers, level shift outliers, and transient shift outliers as anomaly BHP signals. Early detection of abnormal BHP pressure signals can indicate the presence of subsurface fractures, faults, or leaks. Consequently, the correct detection of anomaly points in the pressure signals is of great importance.

Influence of Seasonal Level Shift Outlier on the Residuals of SARMA(1, 1) Model

Time series often contain discrepant observations which are called outliers. Seasonal Level Shift (SLS) outlier is one such which occurs in seasonal time series. And the importance of residuals in time series model building is well known. Thus in this paper, through analytical expressions, we establish the effect of the presence of an SLS outlier on the residuals of the Seasonal Autoregressive Moving Average model of order (1, 1) (SARMA(1, 1)), followed by a simulation study. One of the principle conclusions of this paper is that the SLS outlier not only affects the residuals at its time of incidence but also at the subsequent seasons. And, the extent of the effect depends on the magnitude of the outlier and parameters of the underlying model.. KEYWORDS :Time series, Seasonal level shift, Residuals, Seasonal autoregressive moving average.

ARIMA and TFARIMA Analysis of the Main Water Quality Parameters in the Initial Components of a Megacity’s Drinking Water Supply System

The objective of this paper is to use autoregressive, integrated, and moving average (ARIMA) and transfer function ARIMA (TFARIMA) models to analyze the behavior of the main water quality parameters in the initial components of a drinking water supply system (DWSS) of a megacity (Bogota, Colombia). The DWSS considered in this study consisted of the following components: a river, a reservoir, and a drinking water treatment plant (WTP). Water quality information was collected daily and over a period of 8 years. A comparative analysis was made between the components of the DWSS based on the structure of the ARIMA and TFARIMA models developed. The results show that the best water quality indicators are the following: turbidity > color > total iron. Increasing the time window of the ARIMA analysis (daily/weekly/monthly) suggests an increase in the magnitude of the AR term for each DWSS component (WTP > river > reservoir). This trend suggests that the turbidity behavior in the WTP is more influenced by past observations compared to the turbidity behavior in the river and reservoir, respectively. Smoothing of the data series (moving average) as the time window of the ARIMA analysis increases leads to a greater sensitivity of the model for outlier detection. TFARIMA models suggest that there is no significant influence of past river turbidity events on turbidity in the reservoir, and of reservoir turbidity on turbidity at the WTP outlet. Turbidity outlier events between the river and reservoir occur mainly in a single observation (additive outliers), and between the reservoir and WTP also have a permanent effect over time (level shift outliers). The AR term of the models is useful for studying the transfer of effects between DWSS components, and the MA term is useful for studying the influence of external factors on water quality in each DWSS component.

Diagnostics for partially linear measurement error models

Partially linear models are useful tools to analyze data from economic, genetic, and other fields. Similar to other data analyses, the identification of influential observations that may be potential outliers is an important step beyond estimation in such models. The objective of this article is to develop some diagnostic measures for identifying influential observations in partially linear models when some of the covariates are measured with errors. Deletion measures are developed based on case deletion, mean shift outlier models, and the corrected likelihood of Nakamura (1990). The performance of the methods is illustrated by an artificial example and a real example.

Akurasi Model Hybrid ARIMA-Artificial Neural Network dengan Model Non Hybrid pada Peramalan Peredaran Uang Elektronik di Indonesia

The purpose of this study is to model electronic money in Indonesia using a hybrid model and compare its accuracy with the non-hybrid model. The hybrid model used is Autoregressive Integrated Moving Average (ARIMA)-Artificial Neural Network. The data used is the amount of electronic money circulation for the monthly period January 2009 to October 2021. The ARIMA model formed from research data is ARIMA (1,1,0) with additive outliers and level shift outliers. For Artificial Neural Network modeling is limited by using one hidden layer with three neurons. In the modeling process, 20 repetitions were carried out. The smallest repetition value was obtained, namely the 13th repetition with an error value of 2.569. In this study, it was found that the ARIMA- Artificial Neural Network hybrid model had a smaller Root Mean Squared Error (RMSE) in sample and out sample than the non-hybrid model. Based on the results of the study, it can be concluded that by combining the ARIMA model with Artificial Neural Network, it can increase the accuracy of the data fit results and forecast results.

Indoor Wireless Multipaths Outlier Detection and Clustering

Wireless communication systems have grown and developed significantly in recent years to fulfill the growing demand for high data rates across a wireless medium. Channel models have been used to develop various sturdy wireless systems for indoor and outdoor applications, and these are simulated in the form of datasets. The presence of outliers in clusters has been a concern in datasets, as it affects the standard deviation and mean of the dataset which reduces the data accuracy. In this study, the outliers in the Cooperation in Science and Technology (COST) 2100 MIMO channel model dataset were shifted to the means of the clusters using the Mean Shift Outlier Detection method. Afterward, the data is clustered using simultaneous clustering and model selection matrix affinity (SCAMSMA). The Mean Shift Outlier Detection method identified 52 and 46 multipaths as outliers and improved the clustering accuracy of the indoor scenarios by 3.5% and 0.93%, respectively. It also increased the precision of the clustering based on the decrease in standard deviation of the Jaccard indices from 0.2435 to 0.1807 and 0.3038 to 0.2075.

Stemless components lead to improved radiographic restoration of humeral head anatomy compared with short-stemmed components in total shoulder arthroplasty

Restoring the native center of rotation (COR) in total shoulder arthroplasty (TSA) has been shown to improve postsurgical function, subjective outcomes, and implant longevity. The primary purpose of this study was to compare postoperative radiographic restoration of the humeral COR between short-stem and stemless humeral implants by evaluating the mean COR shift between the 2 techniques. Secondary outcomes evaluated were comparisons of COR shift outliers, humeral head implant thickness and diameter, direction of COR shift, and neck-shaft angle (NSA). This study was a multicenter retrospective comparative study using a consecutive series of primary anatomic TSA patients who received either a short-stem or stemless humeral implant. Radiographically, COR and NSA were measured by 2 fellowship-trained surgeons using the best-fit circle technique on immediate postoperative Grashey radiographs. A total of 229 patients formed the final cohort for analysis that included 89 short stems and 140 stemless components. The mean COR shift for short stems was 2.7 mm (±1.4 mm) compared with 2.1 mm (±0.9 mm) for stemless implants (P<.001). The percentage of short-stem implant patients with a >2 mm COR difference from native was 66.0% (n=62) compared with 47.4% (n=64) for stemless (P=.006). The percentage of short-stem patients with a >4 mm COR difference from native was 17.0% (n=16) compared with 3.0% (n=4) for stemless (P<.001). The mean humeral implant head thickness for short stems was 18.7±2.2 mm compared with 17.2±1.3 mm for stemless implants (P<.001). The mean humeral head diameter for short stems was 48.7±4.4 mm compared with 45.5±3.5 mm for stemless implants (P<.001). The NSA for the short-stem cohort was 136.7° (±3.6°) compared with 133.5° (±6.0°) for stemless (P<.001). Stemless prostheses placed during TSA achieved improved restoration of humeral head COR and were less likely to have significant COR outliers compared with short-stem implants.

Spatial Outlier Accommodation Using a Spatial Variance Shift Outlier Model

Outlier detection has been a long-debated subject among researchers due to its effect on model fitting. Spatial outlier detection has received considerable attention in the recent past. On the other hand, outlier accommodation, particularly in spatial applications, retains vital information about the model. It is pertinent to develop a method that is capable of accommodating detected spatial outliers in a fashion that retains vital information in the spatial models. In this paper, we formulate the variance shift outlier model (SVSOM) in the spatial regression as a robust spatial model using restricted maximum likelihood (REML) and use weights based on the detected outliers in the model. The spatial outliers are accommodated via a revised model for the outlier observations with the help of the SVSOM. Simulation results show that the SVSOM, based on the detected spatial outliers is more efficient than the general spatial model (GSM). The findings of this study also reveal that contamination in the residuals and x variable have little effect on the parameter estimates of the SVSOM, and that outliers in the y variable are always detectable. Asymptotic distribution of the squared spatial prediction residuals are obtained to confirm the outlyingness of an observation. The merit of or proposed SVSOM for the detection and accommodating outliers is also confirmed using artificial and COVID-19 data sets.

Diagnostic measures for the restricted Liu estimator in linear measurement error models

This article concentrated on the diagnostics measures to identify outlier observations using the restricted Liu estimator (RLE) of the vector of parameters in linear measurement error models (LMEMs). To this end, the mean shift outlier model (MSOM) has been determined to detect outliers and the corrected score test statistic. Also, using the RLE, generalized leverage matrices were constructed to identify high-leverage observations. Furthermore, to identify the influential observations, different statistics that are single-case deletion methods are used based on RLE. Finally, a simulation study and a real example are provided to illustrate the performance of the influence measures.

Distance-based directional depth classifiers: a robustness study

Contaminated training sets can highly affect the performance of classification rules. For this reason, robust supervised classifiers have been introduced. Amongst the many, this work focuses on depth-based classifiers, a class of methods which have been proven to enjoy some robustness properties. However, no robustness studies are available for them within a directional data framework. Here, their performance under some directional contamination schemes is evaluated. A comparison with the directional Bayes rule is also provided. Different directional specific contamination scenarios are introduced and discussed: antipodality and orthogonality of the contaminated distribution mean, and the directional mean shift outlier model.

Influence measures and outliers detection in linear mixed measurement error models with Ridge estimation

In this article, we have concentrated on the influence diagnostics based on the mean shift outlier models (MSOM) and case deletion methods in the linear mixed measurement error models with Ridge estimation using the corrected likelihood of Nakamura (1990). We have derived the corrected score test statistic for outliers detection according to MSOM. Then, several case deletion measures are constructed as a tool for influence diagnostics. A parametric bootstrap procedure is used to obtain empirical distributions of the test statistics, and a simulation study has been used to show the performance of the proposed score test statistic and cook distance. Finally, a numerical example is given to illustrate the theoretical results.

A Novel Method of Fault Detection and Identification in a Tightly Coupled, INS/GNSS-Integrated System.

Fault detection and identification are vital for guaranteeing the precision and reliability of tightly coupled inertial navigation system (INS)/global navigation satellite system (GNSS)-integrated navigation systems. A variance shift outlier model (VSOM) was employed to detect faults in the raw pseudo-range data in this paper. The measurements were partially excluded or included in the estimation process depending on the size of the associated shift in the variance. As an objective measure, likelihood ratio and score test statistics were used to determine whether the measurements inflated variance and were deemed to be faulty. The VSOM is appealing because the down-weighting of faulty measurements with the proper weighting factors in the analysis automatically becomes part of the estimation procedure instead of deletion. A parametric bootstrap procedure for significance assessment and multiple testing to identify faults in the VSOM is proposed. The results show that VSOM was validated through field tests, and it works well when single or multiple faults exist in GNSS measurements.

A Bayesian Framework for Robust Quantitative Trait Locus Mapping and Outlier Detection.

We introduce a Bayesian framework for simultaneous feature selection and outlier detection in sparse high-dimensional regression models, with a focus on quantitative trait locus (QTL) mapping in experimental crosses. More specifically, we incorporate the robust mean shift outlier handling mechanism into the multiple QTL mapping regression model and apply LASSO regularization concurrently to the genetic effects and the mean-shift terms through the flexible extended Bayesian LASSO (EBL) prior structure, thereby combining QTL mapping and outlier detection into a single sparse model representation problem. The EBL priors on the mean-shift terms prevent outlying phenotypic values from distorting the genotype-phenotype association and allow their detection as cases with outstanding mean shift values following the LASSO shrinkage. Simulation results demonstrate the effectiveness of our new methodology at mapping QTLs in the presence of outlying phenotypic values and simultaneously identifying the potential outliers, while maintaining a comparable performance to the standard EBL on outlier-free data.

On finite-sample robustness of directional location estimators

Robust location estimators for directional data are known for about 30 years. Scientific literature has focused on studying the asymptotic properties of these estimators like consistency and influence function. Apart from the finite-sample breakdown point, the finite-sample performance of robust directional location estimators has attracted less attention. Hence, it is discussed how the finite-sample max-bias of directional location estimators can be evaluated. Additionally, two new robust estimators of the mean direction are introduced: the spherical Minimum Covariance Determinant estimator (sMCD) and the spherical Minimum Spanning Tree estimator (sMST). The sMCD seeks to identify the densest subset of a given size while the sMST seeks for a well-separated subset. Finally, the robust estimators are compared with respect to the max-bias and to the bias under shift outlier scenarios by means of an extensive simulation study. The results indicate that –in contrast to linear data– the maximum likelihood estimator shows high robustness in terms of the finite-sample max-bias. However, robust estimators are clearly superior to the maximum likelihood estimator in shift outlier contamination schemes.

Use of likelihood ratio tests to detect outliers under the variance shift outlier model

ABSTRACTIn this paper, we revisit the alternative outlier model of Thompson [A note on restricted maximum likelihood estimation with an alternative outlier model, J. Roy. Stat. Soc. Ser. B 47 (1985), pp. 53–55] for detecting outliers in the linear model. Gumedze et al. [A variance shift model for detection of outliers in the linear mixed model, Comput. Statist. Data Anal. 54 (2010), pp. 2128–2144] called this model the variance shift outlier model (VSOM). The basic idea behind the VSOM is to detect observations with inflated variance and isolate them for further investigation. The VSOM is appealing because it downweights an outlier in the analysis, with the weighting determined automatically as part of the estimation procedure. We set up the VSOM as a linear mixed model and then use the likelihood ratio test (LRT) statistic as an objective measure for determining whether the weighting is required, i.e. whether the observation is an outlier. We also derived one-step updates of the variance parameter estimates based on observed, expected and average information matrices to obtain one-step LRT statistics which usually require less computation. Both the fully iterated and one-step LRTs are functions of the squared standard residuals from the null model and therefore can be computed directly without the need to fit the VSOM. We investigated the properties of the likelihood ratio tests and compare them. An extension of the model to detect a group of outliers is also given. We illustrate the proposed methodology using simulated datasets and a real dataset.

Mean shift and influence measures in linear measurement error models with stochastic linear restrictions

ABSTRACTWe present influence diagnostics for linear measurement error models with stochastic linear restrictions using the corrected likelihood of Nakamura in 1990. The case deletion and mean shift outlier models are developed to identify outlying and influential observations. We derive a corrected score test statistic for outlier detection based on mean shift outlier models. The analogs of Cook's distance and likelihood distance are proposed to determine influential observations based on case deletion models. A parametric bootstrap procedure is used to obtain empirical distributions of the test statistics and a simulation study has been used to evaluate the performance of the proposed estimators based on the mean squares error criterion and the score test statistic. Finally, a numerical example is given to illustrate the theoretical results.

Structural Breaks in Potential GDP of Three Major Economies: Just Impaired Credit or the 'New Normal'?

This paper investigates the factors behind the recent growth slowdown (so-called Secular Stagnation) in the US, the euro area and Japan using the metrics of potential output growth. Specifically, our results offer limited support for an impaired credit transmission channel hypothesis (Reinhart and Rogoff, 2009a), while not supporting a supply slowdown hypothesis (Gordon, 2012). We propose a unified framework to test those hypotheses based on structural break tests of potential output. We estimate a variety of potential output growth models accounting for inflation, unemployment, and private credit dynamics (finance-neutral estimates) with multivariate Kalman filters and subject our estimates to structural break tests. We detect structural breaks between 2008 and 2010 for all three countries with Bai-Perron search procedure, the result being robust to the model specification and sample choice, with no significant difference between ordinary and finance-neutral estimates. We proceed with the Chen-Liu test to detect negative temporary change outliers in the Great Recession for the US and the euro area and negative level shift outliers for Japan. Moreover, original breaks in the Chen-Liu test disappear in the US and the euro area once we account for private credit and labour market dynamics, but do not change for Japan.

ANALISIS DATA INFLASI INDONESIA MENGGUNAKAN MODEL AUTOREGRESSIVE INTEGRATED MOVING AVERAGE (ARIMA) DENGAN PENAMBAHAN OUTLIER

The inflation data is one of the financial time series data which often has high volatility. It is caused by the presence of outlier s in the data. Therefore, it is necessary to analyze forecasting that can make all the assumptions are fulled without having to ignore the presence of outlier s. The aim of this study is analyzing the inflation data in Indonesia using ARIMA model with the outlier detection. By modeling annual inflation data in December 2006 to December 2013 there are two types of outlier that are additive outlier (AO) and level shift (LS) outlier . The results show that The ARIMA model with the addition of outlier are better than the ARIMA model without outlier . The ARIMA ([1.12], 1.0) model with the addition of 19 outlier s meet to the all assumptions that are the significance parameters, normality, homoscedasticity, and independence of residuals as well as the smallest MSE value. Keywords: Inflation, ARIMA, Outlier , MSE

Statistical diagnosis for HIV dynamics based on mean shift outlier model

Ordinary differential equation (ODE) are widely used for quantifying HIV viral dynamics. It is interesting but challenging to estimate ODE parameters from noisy data, especially when the data have some outliers. In this study, the authors use the Mean Shift Outlier Model (MSOM) to detect outliers in HIV model based on the two-step estimation of ODE. Approximate formula for shift parameter is derived. Furthermore, a score test statistic is constructed and its approximating distribution is established. The simulation results show that: 1) The boundary points have more impact on the parameter estimation relative to interior points. 2) The proposed procedure can detect the outliers effectively. The authors illustrate the proposed approach using an application example from an HIV clinical trial and find similar pattern to the simulation studies.