Problem Of Outliers Research Articles

Outlier analysis: Natural resources and immigration policy.

This replication underlines the importance of outlier diagnostics since many researchers have long neglected influential observations in OLS regression analysis. In his article, entitled “Primary Resources, Secondary Labor,” Shin finds that advanced democracies with increased natural resource wealth, particularly from oil and natural gas production, are more likely to restrict low-skill immigration policy. By performing outlier diagnostics, this replication shows that Shin’s findings are a statistical artifact. When one outlying country, Norway, is removed from the sample data, I observe almost no significant and negative relationship between oil wealth and immigration policy. When two outlying countries are excluded, the effect of oil wealth completely disappears. Robust regression analysis, a widely used remedial method for outlier problems, confirms the results of my outlier diagnostics.

A new robust parameter estimation approach for multinomial categorical response data with outliers and mismeasured covariates

Being confronted with multinomial categorical response data often occurs in the medical, health, social sciences and other fields. However, the data in practice are subject to measurement error (ME) or outliers due to some unavoidable reasons. There are few studies dealing with problems of ME and outliers in the data with unordered and multi-classified responses simultaneously. To address the effects of ME and outliers in multinomial logistic models, which are most suitable for analyzing categorical outcome variables, we propose novel approaches. Firstly, we develop the extensively weighed corrected score method to estimate parameters in multinomial logistic models with ME. Furthermore, to tackle the problem of that how to obtain parameter estimations when the ME and outliers occur simultaneously, we develop a new robust method, namely the robust extensive weighted corrected score method. Some simulation studies are conducted to evaluate the estimators’ finite sample performance. It has been verified that these estimators are consistent and asymptotically normally distributed under general conditions. They are easy to compute, perform well and stably and are robust against the normality assumption for the ME. The proposed methods are also applied to two real datasets to illustrate the practicability of methods.

LPV Time-Delay System Identification and Its Application to the Centralized Heat-Supply System

In winter, the centralized heat-supply system (CHSS) is an important and indispensable heating equipment in the northeast of China. Hence, in this article, the introduction to the CHSS is first presented, especially for the thermal dynamics of the secondary pipe network (SPN) of the CHSS, and then the identification of the thermal dynamics of the SPN under constant flow is studied. To this end, the linear parameter varying (LPV) model with autoregressive exogenous (ARX) structure is chosen as the system model and the problems of outliers and unknown system delay are both synthesized in the identification process. The physical meanings of all the variables in the LPV model are introduced in detail. By using the expectation-maximization (EM) algorithm, the equations to estimate all the parameters are derived and the desired parameter estimates can be obtained in the context. Finally, the verification tests as well as the corresponding discussions are presented.

The Power of Test of Jennrich Statistic with Robust Methods in Testing the Equality of Correlation Matrices

Jennrich statistic is a method that can be used to test the equality of 2 or more independent correlation matrices. However, Jennrich statistic begins to be problematic when there is presence of outliers that could lead to invalid results. When exiting outliers in data, Jennrich statistic implications will affect Type I errors and will reduce the power of test. To overcome the presence of outliers, this study suggests the use of robust methods as an alternative method and therefore, it will integrate the estimator into Jennrich statistic. Thus, it can improve the testing performance of correlation matrix hypotheses in relation to outlier problems. Therefore, this study proposes 3 statistical tests, namely Js-statistic, Jm-statistic, and Jmad-statistic that can be used to test the equation of 2 or more correlation matrices. The performance of the proposed method is assessed using the power of test. The results show that Jm-statistic and Jmad-statistic can overcome outlier problems into Jennrich statistic in testing the correlation matrix hypothesis. Jmad-statistic is also superior in testing the correlation matrix hypothesis for different sample sizes, especially those involving 10% outliers.

Comparison of M Estimation, S Estimation, with MM Estimation to Get the Best Estimation of Robust Regression in Criminal Cases in Indonesia

Crime incidents that occurred in Indonesia in 2019 based on Survey Based Data on criminal data sourced from the National Socio-Economic Survey and Village Potential Data Collection produced by the Central Statistics Agency recorded 269,324 cases. The high crime rate is caused by several factors, including poverty and population density. Determination of the most influential factors in criminal acts in Indonesia can be done with Regression Analysis. One method of Regression Analysis that is very commonly used is the Least Square Method. However, Regression Analysis can be used if the assumption test is met. If outliers are found, then the assumption test is not completed. The outlier problem can be overcome by using a robust estimation method. This study aims to determine the best estimation method between Maximum Likelihood Type (M) estimation, Scale (S) estimation, and Method of Moment (MM) estimation on Robust Regression. The best estimate of Robust Regression is the smallest Residual Standard Error (RSE) value and the largest Adjusted R-square. The analysis of case studies of criminal acts in Indonesia in 2019 showed that the best estimate was the S estimate with an RSE value of 4226 and an Adjusted R-square of 0.98

New developments in financial statement analysis. Liquidity in the winery sector

The aim of this article is to analyse the short-term solvency of large companies in the wine sector in the period 2014-2018, in two relevant Spanish wine-production areas and assess significant differences in time and between regions. Liquidity is a direct threat to the financial health of companies and is analysed using standard financial indicators and compositional data, in order to prevent the common outlier, non-linearity and asymmetry problems in standard financial ratios. The study shows that the compositional ratios are statistically more adequate and that the turnover indicator between operating cash inflows with respect to current investments and operating cash outflows with respect to current liabilities is a complementary indicator to standard cash flow ratios. Wineries in La Rioja have better liquidity than Catalan wineries in the period under study.

Robust bootstrap regression testing in the presence of outliers

Bootstrap is one of the random sampling methods with replacement, that was proposed to address the problem of small samples whose distributions are difficult to derive. The distribution of bootstrap samples is empirical or free and due to its random sampling with replacement, the probability of choosing a specific observation may be equal to one. Unfortunately, when the original sample data contains an outlier, there is a serious problem that leads to a breakdown OLS (Ordinary Least Squares) estimator, and robust regression methods should be recommended. It is well known that the best robust regression method has a high breakdown point is not more than 0.50, so the robust regression method would break down when the percentage of outliers in the bootstrap sample exceeds 0.50. It is well known that fixed-x bootstrap is resampled the residuals which probably are having outliers. Moreover, the leverage point(s) is an outlier that occurs in X-direction, so the effects of it on fixed-x bootstrap samples would be existence. However, the decision-making about the null hypothesis of bootstrap regression coefficients could not be reliable. In this paper, we propose using weighted fixed-x bootstrap with a probability approach to guarantee the percentage of outliers in the bootstrap samples will be very low. And then weighted M-estimate should be to tackle the problem of outliers and leverage points and taking a more reliable decision about bootstrap regression coefficients hypothesis test. The performance of the suggested method has been tested with others by using real data and simulation. The results show our proposed method is more efficient and reliable than the others.

Modeling of crime rate in Indonesia during the COVID-19 pandemic from a macroeconomic perspective: Using robust regression with S-estimator

Indonesia is the 4th country in Southeast Asia with the highest crime index in 2020. Economic factors are often linked as the main motive for the crime. The purpose of this study is for modeling the crime rate in Indonesia during the Covid-19 pandemic from a macroeconomic perspective. Regression analysis is an analysis that is used to explain and model the relationship between variables. The existence of outliers is often a problem in regression analysis with OLS. To overcome the outlier problem, this research uses robust regression with S-estimation. The results show that was influenced by the unemployment rate, poverty rate, GRDP per capita, population density, and human development index.

A New Approach for Robust Mean-Variance Portfolio Selection Using Trimmed k-Means Clustering

In this study, we consider the data preprocessing using trimmed k-means clustering for robust mean-variance portfolio selection. The proposed method trims the outliers in the data preprocessing stage. The optimum portfolio is formed by selecting the stock representation for each cluster using the Sharpe ratio. The optimum portfolio formation is accomplished by robust fast minimum covariance determinant (FMCD) and robust S mean-variance (MV) portfolio model. In the empirical experiment, we use fundamental trading data for the year 2017 (to form the clusters) and daily closing price data of LQ45 index stocks from August 2017 to July 2018 taken from the Indonesian Stock Exchange to form the optimum portfolio. As benchmark for portfolio performance formed in this study, we use the performance of the Indonesia Composite Index (ICI). The results reveal that the proposed method can reliably obtain the optimum portfolio and solve the outliers problem. Moreover, the comparison stage shows that the combination of trimmed k-means clustering and robust portfolio model is better in forming the optimum portfolio than the combination of k-means clustering and robust MV portfolio model. Finally, we also find that the combination of trimmed k-means clustering (with α=10%) and robust FMCD MV portfolio model outperforms portfolios produced by other methods.

PEMODELAN REGRESI RIDGE ROBUST S,M, MM-ESTIMATOR DALAM PENANGANAN MULTIKOLINIERITAS DAN PENCILAN (Studi Kasus : Faktor-Faktor yang Mempengaruhi Kemiskinan di Jawa Tengah Tahun 2020)

The Ordinary Least Squares (OLS) is one of the most commonly used method to estimate linier regression parameters. If there is a violation of assumptions such as multicolliniearity especially coupled with the outliers, then the regression with OLS is no longer used. One method can be used to solved the multicollinearity and outliers problem is Ridge Robust Regression. Ridge Robust Regression is a modification of ridge regression method used to solve the multicolliniearity and using some estimators of robust regression used to solve the outlier, the estimator including : Maximum likelihood estimator (M-estimator), Scale estimator (S-estimator), and Method of moment estimator (MM-estimator). The case study can be used with this method is data with multicollinearity and outlier, the case study in this research is poverty in Central Java 2020 influenced by life expentancy, unemployment number, GRDP rate, dependency ratio, human development index, the precentage of population over 15 years of age with the highest education in primary school, mean years school. The result of estimation using OLS show that there is a multicollinearity and presence an outliers. Applied the ridge robust regression to case study prove that ridge robust regression can improve parameter estimation. The best ridge robust regression model is Ridge Robust Regression S-Estimator. The influence value of predictor variabels to poverty is 73,08% and the MSE value is 0,00791.

Study of thermal sensation prediction model based on support vector classification (SVC) algorithm with data preprocessing

In order to meet people's demand for comfort, indoor thermal environment often needs to be adjusted. Nevertheless, HVAC target parameters are often over-setted due to cognitive bias, resulting in an uncomfortable environment and waste of energy. Therefore, a thermal sensation prediction model is required to advise the setting of environmental and individual parameters. An accurate mode for predicting thermal sensation encountered severe challenges because of sensor errors, environmental noise, subjective thermal sensation differences, and sample data imbalance. The concrete aim of this paper is to verify the potential of using Support Vector Classification (SVC) algorithm to predict thermal sensation vote (TSV) based on Edited Nearest Neighbour (ENN) and Synthetic Minority Oversampling Technique (SMOTE), named combined ENN + SMOTE + SVC method. Firstly, for the problem of outliers in the dataset, the ENN method was adopted to clean the raw data. Secondly, SMOTE method is used to expand the training data which has the problem of sample imbalance. Finally, SVC algorithm is adopted to build a thermal sensation prediction model. The results show that the model built by the combined ENN + SMOTE + SVC method can achieve better performance than PMV index and other classic classification algorithms.

Object Localization and Tracking System Using Multiple Ultrasonic Sensors with Newton–Raphson Optimization and Kalman Filtering Techniques

In this paper, an object localization and tracking system is implemented with an ultrasonic sensing technique and improved algorithms. The system is composed of one ultrasonic transmitter and five receivers, which uses the principle of ultrasonic ranging measurement to locate the target object. This system has several stages of locating and tracking the target object. First, a simple voice activity detection (VAD) algorithm is used to detect the ultrasonic echo signal of each receiving channel, and then a demodulation method with a low-pass filter is used to extract the signal envelope. The time-of-flight (TOF) estimation algorithm is then applied to the signal envelope for range measurement. Due to the variations of position, direction, material, size, and other factors of the detected object and the signal attenuation during the ultrasonic propagation process, the shape of the echo waveform is easily distorted, and TOF estimation is often inaccurate and unstable. In order to improve the accuracy and stability of TOF estimation, a new method of TOF estimation by fitting the general (GN) model and the double exponential (DE) model on the suitable envelope region using Newton–Raphson (NR) optimization with Levenberg–Marquardt (LM) modification (NRLM) is proposed. The final stage is the object localization and tracking. An extended Kalman filter (EKF) is designed, which inherently considers the interference and outlier problems of range measurement, and effectively reduces the interference to target localization under critical measurement conditions. The performance of the proposed system is evaluated by the experimental evaluation of conditions, such as stationary pen localization, stationary finger localization, and moving finger tracking. The experimental results verify the performance of the system and show that the system has a considerable degree of accuracy and stability for object localization and tracking.

A New Efficient Redescending M-Estimator for Robust Fitting of Linear Regression Models in the Presence of Outliers

Robust regression is an important iterative procedure that seeks analyzing data sets that are contaminated with outliers and unusual observations and reducing their impact over regression coefficients. Robust estimation methods have been introduced to deal with the problem of outliers and provide efficient and stable estimates in their presence. Various robust estimators have been developed in the literature to restrict the unbounded influence of the outliers or leverage points on the model estimates. Here, a new redescending M-estimator is proposed using a novel objective function with the prime focus on getting highly robust and efficient estimates that give promising results. It is evident from the results that, for normal and clean data, the proposed estimator is almost as efficient as ordinary least square method and, however, becomes highly resistant to outliers when it is used for contaminated datasets. The simulation study is being carried out to assess the performance of the proposed redescending M-estimator over different data generation scenarios including normal, t-distribution, and double exponential distributions with different levels of outliers’ contamination, and the results are compared with the existing redescending M-estimators, e.g., Huber, Tukey Biweight, Hampel, and Andrew-Sign function. The performance of the proposed estimators was also checked using real-life data applications of the estimators and found that the proposed estimators give promising results as compared to the existing estimators.

Lithium-ion battery capacity estimation based on open circuit voltage identification using the iteratively reweighted least squares at different aging levels

In traditional incremental SOC (stage of charge)-capacity method, errors of the selected SOC points inevitably lead to inaccurate capacity estimation. In this paper, a capacity estimation method based on OCV (open circuit voltage) identification and IRLS (iteratively reweighted least squares) algorithm method is proposed. Firstly, the accuracy and complexity of three parameter identification schemes of first-order RC model at different aging levels are studied with the forgetting factor recursive least squares algorithm. Through the online identified OCV by the optimal scheme, the full interval SOC estimation can be obtained indirectly. Finally, the capacity is estimated using the full interval SOC estimation with IRLS algorithm, which avoids the randomness of the SOC point selection, and solves the problem of possible outliers in the SOC estimation or the unknown variance of the random error of the SOC gain. Through the designed aging experiment, the capacity estimation accuracy is verified under different life conditions. The results show that the estimation accuracy of this method is high and the error does not exceed 2.5%.

A probabilistic generalization of isolation forest

The problem of finding anomalies and outliers in datasets is one of the most important challenges of modern data analysis. Among the commonly dedicated tools to solve this task one can find Isolation Forest (IF) that is an efficient, conceptually simple, and fast method. In this study, we propose the Probabilistic Generalization of Isolation Forest (PGIF) that is an intuitively appealing and efficient enhancement of the original approach. The proposed generalization is based on nonlinear dependence of segment-cumulated probability from the length of segment. Introduction of the generalization allows to achieve more effective splits that are rather performed between the clusters, i.e. regions where datapoints constitute dense formations and not through them. In a comprehensive series of experiments, we show that the proposed method allows us to detect anomalies hidden between clusters more effectively. Moreover, it is demonstrated that our approach favorably affects the quality of anomaly detection in both artificial and real datasets. In terms of time complexity our method is close to the original one since the generalization is related only to the building of the trees while the scoring procedure (which takes the main time) is kept unchanged.

Modified Robust Ridge M-Estimators in Two-Parameter Ridge Regression Model

The methods of two-parameter ridge and ordinary ridge regression are very sensitive to the presence of the joint problem of multicollinearity and outliers in the y-direction. To overcome this problem, modified robust ridge M-estimators are proposed. The new estimators are then compared with the existing ones by means of extensive Monte Carlo simulations. According to mean squared error (MSE) criterion, the new estimators outperform the least square estimator, ridge regression estimator, and two-parameter ridge estimator in many considered scenarios. Two numerical examples are also presented to illustrate the simulation results.

Mean Estimators Using Robust Quantile Regression and L-Moments’ Characteristics for Complete and Partial Auxiliary Information

Ratio type regression estimator is a prevalent and readily implemented heuristic under simple random sampling (SRS) and two-stage sampling for the estimation of population. But this existing method is based on the ordinary least square (OLS) regression coefficient which is not an effective approach in the presence outliers in the data. In this article, we proposed a class of estimators firstly for complete auxiliary information and, later on, for partial auxiliary information for the presence of outliers in the data. To address this problem, initially we presented a distinct class of estimators by introducing the characteristics of L-moments in the existing estimators. Later on, quantile regression estimators are defined as more robust in the presence of outliers. These techniques empowered the proposed estimators to handle the problem of outliers. To prove the better performance of the proposed estimators, numerical studies are carried out using R language. To calculate the mean square error (MSE), hypothetical equations are expressed for adapted and proposed estimators. Percentage Relative Efficiencies (PRE) are compared to justify the proposed estimators.

Using Correlation Coefficient to Solve Outliers Problem in Regression Analysis, with Practical Application

Using Correlation Coefficient to Solve Outliers Problem in Regression Analysis, with Practical Application

Robust Dawoud–Kibria estimator for handling multicollinearity and outliers in the linear regression model

In the linear regression model, least-squares (LS) estimator is usually used for estimating regression parameters. LS is an unreliable and unfavourable estimator when multicollinearity and outlier problems exist in the model. Therefore, we propose a new robust regression estimator for solving the abovementioned problems simultaneously. We conducted theoretical comparisons and different scenarios of simulation studies, and a real-life dataset was employed to show the performance of the proposed estimator. Results showed that the proposed estimator performs better than other estimators when multicollinearity and outlier problems occur simultaneously in the model.

Kernel weighted least square approach for imputing missing values of metabolomics data

Mass spectrometry is a modern and sophisticated high-throughput analytical technique that enables large-scale metabolomic analyses. It yields a high-dimensional large-scale matrix (samples × metabolites) of quantified data that often contain missing cells in the data matrix as well as outliers that originate for several reasons, including technical and biological sources. Although several missing data imputation techniques are described in the literature, all conventional existing techniques only solve the missing value problems. They do not relieve the problems of outliers. Therefore, outliers in the dataset decrease the accuracy of the imputation. We developed a new kernel weight function-based proposed missing data imputation technique that resolves the problems of missing values and outliers. We evaluated the performance of the proposed method and other conventional and recently developed missing imputation techniques using both artificially generated data and experimentally measured data analysis in both the absence and presence of different rates of outliers. Performances based on both artificial data and real metabolomics data indicate the superiority of our proposed kernel weight-based missing data imputation technique to the existing alternatives. For user convenience, an R package of the proposed kernel weight-based missing value imputation technique was developed, which is available at https://github.com/NishithPaul/tWLSA.