Heavy-tailed Errors Research Articles

Robust regression for optimal individualized treatment rules.

Because different patients may respond quite differently to the same drug or treatment, there is an increasing interest in discovering individualized treatment rules. In particular, there is an emerging need to find optimal individualized treatment rules, which would lead to the "best" clinical outcome. In this paper, we propose a new class of loss functions and estimators based on robust regression to estimate the optimal individualized treatment rules. Compared to existing estimation methods in the literature, the new estimators are novel and advantageous in the following aspects. First, they are robust against skewed, heterogeneous, heavy-tailed errors or outliers in data. Second, they are robust against a misspecification of the baseline function. Third, under some general situations, the new estimator coupled with the pinball loss approximately maximizes the outcome's conditional quantile instead of the conditional mean, which leads to a more robust optimal individualized treatment rule than the traditional mean-based estimators. Consistency and asymptotic normality of the proposed estimators are established. Their empirical performance is demonstrated via extensive simulation studies and an analysis of an AIDS data set.

MIXED CAUSAL-NONCAUSAL AR PROCESSES AND THE MODELLING OF EXPLOSIVE BUBBLES

Noncausal autoregressive models with heavy-tailed errors generate locally explosive processes and, therefore, provide a convenient framework for modelling bubbles in economic and financial time series. We investigate the probability properties of mixed causal-noncausal autoregressive processes, assuming the errors follow a stable non-Gaussian distribution. Extending the study of the noncausal AR(1) model by Gouriéroux and Zakoian (2017), we show that the conditional distribution in direct time is lighter-tailed than the errors distribution, and we emphasize the presence of ARCH effects in a causal representation of the process. Under the assumption that the errors belong to the domain of attraction of a stable distribution, we show that a causal AR representation with non-i.i.d. errors can be consistently estimated by classical least-squares. We derive a portmanteau test to check the validity of the estimated AR representation and propose a method based on extreme residuals clustering to determine whether the AR generating process is causal, noncausal, or mixed. An empirical study on simulated and real data illustrates the potential usefulness of the results.

Robust Bayesian seemingly unrelated regression model

A robust Bayesian model for seemingly unrelated regression is proposed. By using heavy-tailed distributions for the likelihood, robustness in the response variable is attained. In addition, this robust procedure is combined with a diagnostic approach to identify observations that are far from the bulk of the data in the multivariate space spanned by all variables. The most distant observations are downweighted to reduce the effect of leverage points. The resulting robust Bayesian model can be interpreted as a heteroscedastic seemingly unrelated regression model. Robust Bayesian estimates are obtained by a Markov Chain Monte Carlo approach. Complications by using a heavy-tailed error distribution are resolved efficiently by representing these distributions as a scale mixture of normal distributions. Monte Carlo simulation experiments confirm that the proposed model outperforms its traditional Bayesian counterpart when the data are contaminated in the response and/or the input variables. The method is demonstrated on a real dataset.

Model averaging for M-estimation

ABSTRACTM-estimation is a widely used technique for robust statistical inference. In this paper, we study model selection and model averaging for M-estimation to simultaneously improve the coverage probability of confidence intervals of the parameters of interest and reduce the impact of heavy-tailed errors or outliers in the response. Under general conditions, we develop robust versions of the focused information criterion and a frequentist model average estimator for M-estimation, and we examine their theoretical properties. In addition, we carry out extensive simulation studies as well as two real examples to assess the performance of our new procedure, and find that the proposed method produces satisfactory results.

Multivariate calibration with robust signal regression

Motivated by a multivariate calibration problem from a soil characterization study, we proposed tractable and robust variants of penalized signal regression (PSR) using a class of non-convex Huber-like criteria as the loss function. Standard methods may fail to produce a reliable estimator, especially when there are heavy-tailed errors. We present a computationally efficient algorithm to solve this non-convex problem. Simulation and empirical examples are extremely promising and show that the proposed algorithm substantially improves the PSR performance under heavy-tailed errors.

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates

In this paper, a new robust and efficient estimation approach based on local modal regression is proposed for partially linear models with large-dimensional covariates. We show that the resulting estimators for both parametric and nonparametric components are more efficient in the presence of outliers or heavy-tail error distribution, and as asymptotically efficient as the corresponding least squares estimators when there are no outliers and the error distribution is normal. We also establish the asymptotic properties of proposed estimators when the covariate dimension diverges at the rate of $$o\left( {\sqrt{n} } \right) \mathrm{{ }}$$ . To achieve sparsity and enhance interpretability, we develop a variable selection procedure based on SCAD penalty to select significant parametric covariates and show that the method enjoys the oracle property under mild regularity conditions. Moreover, we propose a practical modified MEM algorithm for the proposed procedures. Some Monte Carlo simulations and a real data are conducted to illustrate the finite sample performance of the proposed estimators. Finally, based on the idea of sure independence screening procedure proposed by Fan and Lv (J R Stat Soc 70:849–911, 2008), a robust two-step approach is introduced to deal with ultra-high dimensional data.

Robust exponential squared loss-based estimation in semi-functional linear regression models

In this paper, we present a new robust estimation procedure for semi-functional linear regression models by using exponential squared loss. The outstanding advantage of the proposed method is the resulting estimators are more efficient than the least squares estimators in the presence of outliers or heavy-tail error distributions. The slope function and functional predictor variable are approximated by functional principal component basis functions. Under some regularity conditions, we obtain the optimal convergence rate of slope function, and the asymptotic normality of parameter vector and variance estimator. Finally, we investigate the finite sample performance of the proposed method through a simulation study and real data analysis.

Shrinkage and penalized estimators in weighted least absolute deviations regression models

ABSTRACTIn this paper, we consider the estimation problem of the weighted least absolute deviation (WLAD) regression parameter vector when there are some outliers or heavy-tailed errors in the response and the leverage points in the predictors. We propose the pretest and James–Stein shrinkage WLAD estimators when some of the parameters may be subject to certain restrictions. We derive the asymptotic risk of the pretest and shrinkage WLAD estimators and show that if the shrinkage dimension exceeds two, the asymptotic risk of the shrinkage WLAD estimator is strictly less than the unrestricted WLAD estimator. On the other hand, the risk of the pretest WLAD estimator depends on the validity of the restrictions on the parameters. Furthermore, we study the WLAD absolute shrinkage and selection operator (WLAD-LASSO) and compare its relative performance with the pretest and shrinkage WLAD estimators. A simulation study is conducted to evaluate the performance of the proposed estimators relative to that of the unrestricted WLAD estimator. A real-life data example using body fat study is used to illustrate the performance of the suggested estimators.

Inference of high-dimensional linear models with time-varying coefficients

We propose a pointwise inference algorithm for high-dimensional linear models with time-varying coefficients. The method is based on a novel combination of the nonparametric kernel smoothing technique and a Lasso bias-corrected ridge regression estimator. Due to the non-stationarity feature of the model, dynamic bias-variance decomposition of the estimator is obtained. With a bias-correction procedure, the local null distribution of the estimator of the time-varying coefficient vector is characterized for iid Gaussian and heavy-tailed errors. The limiting null distribution is also established for Gaussian process errors, and we show that the asymptotic properties differ between short-range and long-range dependent errors. Here, p-values are adjusted by a Bonferroni-type correction procedure to control the familywise error rate (FWER) in the asymptotic sense at each time point. The finite sample size performance of the proposed inference algorithm is illustrated with synthetic data and an application to learn brain connectivity by using the resting-state fMRI data for Parkinson's disease.

M-estimation and model identification based on double SCAD penalization

ABSTRACTM-estimation is a widely used method for robust statistical inference. In this article, using a B-spline series approximation with a double smoothly clipped absolute deviation penalization, we solve the problem of simultaneous variable selection and parametric component identification in a non parametric additive model. The theoretical properties of the double non concave penalized M-estimation are established. The proposed approach is resistant to heavy-tailed errors or outliers in the responses. Simulation studies for finite-sample cases are conducted and a real dataset is also analyzed for illustration of this new approach.

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING.

Heavy-tailed errors impair the accuracy of the least squares estimate, which can be spoiled by a single grossly outlying observation. As argued in the seminal work of Peter Huber in 1973 [Ann. Statist.1 (1973) 799-821], robust alternatives to the method of least squares are sorely needed. To achieve robustness against heavy-tailed sampling distributions, we revisit the Huber estimator from a new perspective by letting the tuning parameter involved diverge with the sample size. In this paper, we develop nonasymptotic concentration results for such an adaptive Huber estimator, namely, the Huber estimator with the tuning parameter adapted to sample size, dimension, and the variance of the noise. Specifically, we obtain a sub-Gaussian-type deviation inequality and a nonasymptotic Bahadur representation when noise variables only have finite second moments. The nonasymptotic results further yield two conventional normal approximation results that are of independent interest, the Berry-Esseen inequality and Cramér-type moderate deviation. As an important application to large-scale simultaneous inference, we apply these robust normal approximation results to analyze a dependence-adjusted multiple testing procedure for moderately heavy-tailed data. It is shown that the robust dependence-adjusted procedure asymptotically controls the overall false discovery proportion at the nominal level under mild moment conditions. Thorough numerical results on both simulated and real datasets are also provided to back up our theory.

WLAD-LASSO method for robust estimation and variable selection in partially linear models

ABSTRACTThis paper focuses on robust estimation and variable selection for partially linear models. We combine the weighted least absolute deviation (WLAD) regression with the adaptive least absolute shrinkage and selection operator (LASSO) to achieve simultaneous robust estimation and variable selection for partially linear models. Compared with the LAD-LASSO method, the WLAD-LASSO method will resist to the heavy-tailed errors and outliers in the parametric components. In addition, we estimate the unknown smooth function by a robust local linear regression. Under some regular conditions, the theoretical properties of the proposed estimators are established. We further examine finite-sample performance of the proposed procedure by simulation studies and a real data example.

A nonparametric test for ordered nested effects

We develop a nonparametric procedure for a nested design that tests for increasing nested factor levels. Within this context, the exact distribution, the exact conditional distribution that adjusts for ties, and the asymptotic distribution are discussed. Type I error rates and power estimates were computed via simulation for several nested designs. The simulations show that the proposed procedure has stable Type I error rates and outperforms a parametric version of the test for heavy-tailed error distributions. For the sake of illustration, we also analyze the data set that spawned the necessity for the proposed procedure.

Local influence for elliptical partially varying-coefficient model

In this article, we extend varying-coefficient models with normal errors to elliptical errors in order to permit distributions with heavier and lighter tails than the normal ones. This class of models includes all symmetric continuous distributions, such as Student-t, Pearson VII, power exponential and logistic, among others. Estimation is performed by maximum penalized likelihood method and by using smoothing splines. In order to study the sensitivity of the penalized estimates under some usual perturbation schemes in the model or data, the local influence curvatures are derived and some diagnostic graphics are proposed. A real dataset previously analysed by using varying-coefficient models with normal errors is reanalysed under varying-coefficient models with heavy-tailed errors.

Efficient Robust Estimation for Linear Models with Missing Response at Random.

Coefficient estimation in linear regression models with missing data is routinely done in the mean regression framework. However, the mean regression theory breaks down if the error variance is infinite. In addition, correct specification of the likelihood function for existing imputation approach is often challenging in practice, especially for skewed data. In this paper, we develop a novel composite quantile regression and a weighted quantile average estimation procedure for parameter estimation in linear regression models when some responses are missing at random. Instead of imputing the missing response by randomly drawing from its conditional distribution, we propose to impute both missing and observed responses by their estimated conditional quantiles given the observed data and to use the parametrically estimated propensity scores to weigh check functions that define a regression parameter. Both estimation procedures are resistant to heavy-tailed errors or outliers in the response and can achieve nice robustness and efficiency. Moreover, we propose adaptive penalization methods to simultaneously select significant variables and estimate unknown parameters. Asymptotic properties of the proposed estimators are carefully investigated. An efficient algorithm is developed for fast implementation of the proposed methodologies. We also discuss a model selection criterion, which is based on an IC Q -type statistic, to select the penalty parameters. The performance of the proposed methods is illustrated via simulated and real data sets.

A descent method for least absolute deviation lasso problems

Variable selection is an important method to analyze large quantity of data and extract useful information. Although least square regression is the most widely used scheme for its flexibility in obtaining explicit solutions, least absolute deviation (LAD) regression combined with lasso penalty becomes popular for its resistance to heavy-tailed errors in response variable, denoted as LAD-LASSO. In this paper, we consider the LAD-LASSO problem for variable selection. Based on a dynamic optimality condition of nonsmooth optimization problem, we develop a descent method to solve the nonsmooth optimization problem. Numerical experiments are conducted to confirm that the proposed method is more efficient than existing methods.

Robust change point detection method via adaptive LAD-LASSO

Change point problem is one of the hot issues in statistics, econometrics, signal processing and so on. LAD estimator is more robust than OLS estimator, especially when datasets subject to heavy tailed errors or outliers. LASSO is a popular choice for shrinkage estimation. In the paper, we combine the two classical ideas together to put forward a robust detection method via adaptive LAD-LASSO to estimate change points in the mean-shift model. The basic idea is converting the change point estimation problem into variable selection problem with penalty. An enhanced two-step procedure is proposed. Simulation and a real example show that the novel method is really feasible and the fast and effective computation algorithm is easier to realize.

Adaptive robust estimation in joint mean–covariance regression model for bivariate longitudinal data

ABSTRACTThe estimation of the covariance matrix is important in the analysis of bivariate longitudinal data. A good estimator for the covariance matrix can improve the efficiency of the estimators of the mean regression coefficients. Furthermore, the covariance estimation itself is also of interest, but it is a challenging job to model the covariance matrix of bivariate longitudinal data due to the complex structure and positive definite constraint. In addition, most of existing approaches are based on the maximum likelihood, which is very sensitive to outliers or heavy-tail error distributions. In this article, an adaptive robust estimation method is proposed for bivariate longitudinal data. Unlike the existing likelihood-based methods, the proposed method can adapt to different error distributions. Specifically, at first, we utilize the modified Cholesky block decomposition to parameterize the covariance matrices. Secondly, we apply the bounded Huber's score function to develop a set of robust generalized estimating equations to estimate the parameters both in the mean and the covariance models simultaneously. A data-driven approach is presented to select the parameter c in the Huber's score function, which can ensure that the proposed method is robust and efficient. A simulation study and a real data analysis are conducted to illustrate the robustness and efficiency of the proposed approach.

Elliptical linear mixed models with a covariate subject to measurement error

In this paper we extend linear mixed models with elliptical errors by adding a covariate subject to measurement error in the linear predictor. The former class is defined appropriately so that the joint marginal distribution of the response and the observed covariate subject to measurement error is also elliptical. Thus, numerical integration methods are not required to obtain the marginal model and the mean and the variance–covariance structures of the hierarchical model are preserved. A kurtosis flexibility is allowed for each joint marginal distribution and since the conditional distributions are also elliptical, the predictions of the random effects as well as of the covariate subject to measurement error may be performed in a similar way of the normal case. A reweighed iterative process based on the maximum likelihood method is derived for obtaining the parameter estimates, which appear to be robust against outlying observations in the sense of the Mahalanobis distance. In order to assess the sample properties of the maximum likelihood parameter estimates as well as their asymptotic standard errors, a simulation study is performed under different parameter settings and error distributions. Goodness-of-fit measures based on the Mahalanobis distance are presented and normal curvatures of local influence are derived under three usual perturbation schemes, which are selected appropriately. Finally, an illustrative example previously analyzed under normal error models is reanalyzed by considering heavy-tailed error models. The diagnostic procedures are applied for comparing the fitted models.

Variable selection in partially linear additive models for modal regression

ABSTRACTBased on B-spline basis functions and smoothly clipped absolute deviation (SCAD) penalty, we present a new estimation and variable selection procedure based on modal regression for partially linear additive models. The outstanding merit of the new method is that it is robust against outliers or heavy-tail error distributions and performs no worse than the least-square-based estimation for normal error case. The main difference is that the standard quadratic loss is replaced by a kernel function depending on a bandwidth that can be automatically selected based on the observed data. With appropriate selection of the regularization parameters, the new method possesses the consistency in variable selection and oracle property in estimation. Finally, both simulation study and real data analysis are performed to examine the performance of our approach.