Asymptotic Risk Research Articles

On the asymptotic risk of ridge regression with many predictors

On the asymptotic risk of ridge regression with many predictors

Phase transition and higher order analysis of Lq regularization under dependence.

We study the problem of estimating a [Formula: see text]-sparse signal [Formula: see text] from a set of noisy observations [Formula: see text] under the model [Formula: see text], where [Formula: see text] is the measurement matrix the row of which is drawn from distribution [Formula: see text]. We consider the class of [Formula: see text]-regularized least squares (LQLS) given by the formulation [Formula: see text], where [Formula: see text] [Formula: see text] denotes the [Formula: see text]-norm. In the setting [Formula: see text] with fixed [Formula: see text] and [Formula: see text], we derive the asymptotic risk of [Formula: see text] for arbitrary covariance matrix [Formula: see text] that generalizes the existing results for standard Gaussian design, i.e. [Formula: see text]. The results were derived from the non-rigorous replica method. We perform a higher-order analysis for LQLS in the small-error regime in which the first dominant term can be used to determine the phase transition behavior of LQLS. Our results show that the first dominant term does not depend on the covariance structure of [Formula: see text] in the cases [Formula: see text] and [Formula: see text] which indicates that the correlations among predictors only affect the phase transition curve in the case [Formula: see text] a.k.a. LASSO. To study the influence of the covariance structure of [Formula: see text] on the performance of LQLS in the cases [Formula: see text] and [Formula: see text], we derive the explicit formulas for the second dominant term in the expansion of the asymptotic risk in terms of small error. Extensive computational experiments confirm that our analytical predictions are consistent with numerical results.

Stein-type control function maximum likelihood estimator for the probit model in the presence of endogeneity

A Stein-type control function maximum likelihood (CFML) estimator is suggested for the probit model in the presence of endogeneity. This novel estimator combines the probit maximum likelihood and CFML estimators. The asymptotic distribution and risk function for the new estimator is derived. It is demonstrated that, subject to certain conditions of the shrinkage parameter, the asymptotic risk of the new estimator is strictly smaller than the risk of the CFML. Monte Carlo simulations illustrate the method's superiority in finite samples. The method is also applied to analyze the impact of managerial incentives on the use of foreign-exchange derivatives.

A weighted average limited information maximum likelihood estimator

In this article, a Stein-type weighted limited information maximum likelihood (LIML) estimator is proposed. It is based on a weighted average of the ordinary least squares (OLS) and LIML estimators, with weights inversely proportional to the Hausman test statistic. The asymptotic distribution of the proposed estimator is derived by means of local-to-exogenous asymptotic theory. In addition, the asymptotic risk of the Stein-type LIML estimator is calculated, and it is shown that the risk is strictly smaller than the risk of the LIML under certain conditions. A Monte Carlo simulation and an empirical application of a green patent dataset from Nordic countries are used to demonstrate the superiority of the Stein-type LIML estimator to the OLS, two-stage least squares, LIML and combined estimators when the number of instruments is large.

Locally adaptive Bayesian isotonic regression using half shrinkage priors

AbstractIsotonic regression or monotone function estimation is a problem of estimating function values under monotonicity constraints, which appears naturally in many scientific fields. This paper proposes a new Bayesian method with global–local shrinkage priors for estimating monotone function values. Specifically, we introduce half shrinkage priors for positive valued random variables and assign them for the first‐order differences of function values. We also develop fast and simple Gibbs sampling algorithms for full posterior analysis. By incorporating advanced shrinkage priors, the proposed method is adaptive to local abrupt changes or jumps in target functions. We show this adaptive property theoretically by proving that the posterior mean estimators are robust to large differences and that asymptotic risk for unchanged points can be improved. Finally, we demonstrate the proposed methods through simulations and applications to a real data set.

Structural breaks in seemingly unrelated regression models

AbstractThis paper develops an efficient Stein-like shrinkage estimator for estimating slope parameters under structural breaks in seemingly unrelated regression models, which is then used for forecasting. The proposed method is a weighted average of two estimators: a restricted estimator that estimates the parameters under the restriction of no break in the coefficients, and an unrestricted estimator that considers break points and estimates the parameters using the observations within each regime. It is established that the asymptotic risk of the Stein-like shrinkage estimator is smaller than that of the unrestricted estimator, which is the method typically used to estimate the slope coefficients under structural breaks. Furthermore, this paper proposes an averaging minimal mean squared error estimator in which the averaging weight is derived by minimizing its asymptotic risk. Insights from the theoretical analysis are demonstrated in Monte Carlo simulations and through an empirical example of forecasting output growth of G7 countries.

Forecasting vector autoregressions with mixed roots in the vicinity of unity

This article evaluates the forecast performance of model averaging forecasts in a nonstationary vector autoregression with mixed roots in the vicinity of unity. The deviation from unit root allows for local to unity, moderate deviation from unity and strong unit root, and the direction of such deviation could be from either the stationary or the explosive side. We provide a theoretical foundation for comparison among various forecasts, including the least squares estimator, the constrained estimator imposing the unit root constraint, and the selection or average over these two basic estimators. Furthermore, three new types of estimators are constructed, i.e., the bagging versions of the pretest estimator, the Mallows-pretest estimator that marries the Mallows averaging criterion and the Wald test, and the Mallows-bagging estimator that combines the Mallows averaging criterion and bagging technique. The asymptotic risks are shown to depend on the local parameters, which are not consistently estimable. Via Monte Carlo simulations, graphic comparisons indicate that the Mallows averaging estimator has both robust and outstanding forecasting performance. Model averaging over the vector autoregressive lag order is further considered to address the issue of model uncertainty in the lag specification. Finite sample simulations show that the Mallows averaging estimator performs superior to other frequently used selection and averaging methods. The application to forecasting the financial indices popularly used in the predictive regression further illustrates the practical merit of the proposed estimator.

Triple Sampling Inference Procedures for the Mean of the Normal Distribution When the Population Coefficient of Variation Is Known

This paper discusses the triple sampling inference procedures for the mean of a symmetric distribution—the normal distribution when the coefficient of variation is known. We use the Searls’ estimator as an initial estimate for the unknown population mean rather than the classical sample mean. In statistics literature, the normal distribution under investigation underlines almost all the natural phenomena with applications in many fields. First, we discuss the minimum risk point estimation problem under a squared error loss function with linear sampling cost. We obtained all asymptotic results that enhanced finding the second-order asymptotic risk and regret. Second, we construct a fixed-width confidence interval for the mean that satisfies at least a predetermined nominal value and find the second-order asymptotic coverage probability. Both estimation problems are performed under a unified optimal framework. The theoretical results reveal that the performance of the triple sampling procedure depends on the numerical value of the coefficient of variation—the smaller the coefficient of variation, the better the performance of the procedure.

Pretest and shrinkage estimators in generalized partially linear models with application to real data

AbstractSemiparametric models hold promise to address many challenges to statistical inference that arise from real‐world applications, but their novelty and theoretical complexity create challenges for estimation. Taking advantage of the broad applicability of semiparametric models, we propose some novel and improved methods to estimate the regression coefficients of generalized partially linear models (GPLM). This model extends the generalized linear model by adding a nonparametric component. Like in parametric models, variable selection is important in the GPLM to single out the inactive covariates for the response. Instead of deleting inactive covariates, our approach uses them as auxiliary information in the estimation procedure. We then define two models, one that includes all the covariates and another that includes the active covariates only. We then combine these two model estimators optimally to form the pretest and shrinkage estimators. Asymptotic properties are studied to derive the asymptotic biases and risks of the proposed estimators. We show that if the shrinkage dimension exceeds two, the asymptotic risks of the shrinkage estimators are strictly less than those of the full model estimators. Extensive Monte Carlo simulation studies are conducted to examine the finite‐sample performance of the proposed estimation methods. We then apply our proposed methods to two real data sets. Our simulation and real data results show that the proposed estimators perform with higher accuracy and lower variability in the estimation of regression parameters for GPLM compared with competing estimation methods.

Restricted Stein-rule estimation in ultrastructural linear measurement error models

We consider a multivariate ultrastructural measurement error model when some information on regression coefficients is available in the form of linear restriction. We propose Stein-rule estimation for the regression coefficients. Without any assumption about the distribution for the random components, the asymptotic risk under a specific quadratic loss function is studied and a sufficient condition for the dominance of the proposed estimator over a consistent estimator. We also carry out a simulation study to demonstrate the finite sample properties of the estimators.

Asymptotic risk decomposition for regularly varying distributions with tail dependence

In this paper we investigate the limiting behaviour of Conditional Tail Expectation (CTE) and its decomposition for a sum of real-valued tail-dependent random variables with regularly varying distributions. Asymptotic proportions to the corresponding Value at Risk (VaR) measures are obtained for a flexible dependence structure. For a certain practical case considering an investment portfolio exact formulas are derived and sensitivity to model parameters is analysed. We also carry out a simulation study verifying our results and revealing the speed of convergence for different values of parameters.

Efficient estimation method for generalized ARFIMA models

This paper focuses on pretest and shrinkage estimation strategies for generalized autoregressive fractionally integrated moving average (GARFIMA) models when some of the regression parameters are possible to restrict to a subspace. These estimation strategies are constructed on the assumption that some covariates are not statistically significant for the response. To define the pretest and shrinkage estimators, we fit two models: one includes all the covariates and the others are subject to linear constraint based on the auxiliary information of the insignificant covariates. The unrestricted and restricted estimators are then combined optimally to get the pretest and shrinkage estimators. We enlighten the statistical properties of the shrinkage and pretest estimators in terms of asymptotic bias and risk. We examine the comparative performance of pretest and shrinkage estimators with respect to unrestricted maximum partial likelihood estimator (UMPLE). We show that the shrinkage estimators have a lower relative mean squared error as compared to the UMPLE when the number of significant covariates exceeds two. Monte Carlo simulations are conducted to examine the relative performance of the proposed estimators to the UMPLE. An empirical application is used for the usefulness of our proposed estimation strategies.

Relaxing the Assumption of Conditional Independence in a Asymptotic Single Risk Factor Model

Relaxing the Assumption of Conditional Independence in a Asymptotic Single Risk Factor Model

LASSO risk and phase transition under dependence

We consider the problem of recovering a k-sparse signal β0∈Rp from noisy observations y=Xβ0+w∈Rn. One of the most popular approaches is the l1-regularized least squares, also known as LASSO. We analyze the mean square error of LASSO in the case of random designs in which each row of X is drawn from distribution N(0,Σ) with general Σ. We first derive the asymptotic risk of LASSO for w≠0 in the limit of n,p→∞ with n∕p→δ∈[0,∞). We then examine conditions on n, p, and k for LASSO to exactly reconstruct β0 in the noiseless case w=0. A phase boundary δc=δ(ϵ) is precisely established in the phase space defined by 0≤δ,ϵ≤1, where ϵ=k∕p. Above this boundary, LASSO perfectly recovers β0 with high probability. Below this boundary, LASSO fails to recover β0 with high probability. While the values of the non-zero elements of β0 do not have any effect on the phase transition curve, our analysis shows that δc does depend on the signed pattern of the nonzero values of β0 for general Σ≠Ip×p. This is in sharp contrast to the previous phase transition results derived in i.i.d. case with Σ= Ip×p where δc is completely determined by ϵ regardless of the distribution of β0. Underlying our formalism is a recently developed efficient algorithm called approximate message passing (AMP) algorithm. We generalize the state evolution of AMP from i.i.d. case to general case with Σ≠Ip×p. Extensive computational experiments confirm that our theoretical predictions are consistent with simulation results on moderate size system.

Shrinkage estimation of panel data models with interactive effects

This study proposes a shrinkage estimator for linear panel data models. Our estimator is a weighted average of the additive and interactive effects estimators where the weight is inversely proportional to the Hausman-type test statistic. We establish the asymptotic distribution of the shrinkage estimator and show that its asymptotic risk is strictly smaller than that of the interactive estimator. A Monte Carlo simulation shows that the shrinkage estimator has a substantially smaller finite sample error relative to other estimators.

Optimal Shrinkage Estimation of Predictive Densities Under α-Divergences

We consider the problem of estimating the predictive density in a heteroskedastic Gaussian model under general divergence loss. Based on a conjugate hierarchical set-up, we consider generic classes of shrinkage predictive densities that are governed by location and scale hyper-parameters. For any α-divergence loss, we propose a risk-estimation based methodology for tuning these shrinkage hyper-parameters. Our proposed predictive density estimators enjoy optimal asymptotic risk properties that are in concordance with the optimal shrinkage calibration point estimation results established by Xie, Kou, and Brown (2012) for heteroskedastic hierarchical models. These α-divergence risk optimality properties of our proposed predictors are not shared by empirical Bayes predictive density estimators that are calibrated by traditional methods such as maximum likelihood and method of moments. We conduct several numerical studies to compare the non-asymptotic performance of our proposed predictive density estimators with other competing methods and obtain encouraging results.

Sparse Estimation Strategies in Linear Mixed Effect Models for High-Dimensional Data Application.

In a host of business applications, biomedical and epidemiological studies, the problem of multicollinearity among predictor variables is a frequent issue in longitudinal data analysis for linear mixed models (LMM). We consider an efficient estimation strategy for high-dimensional data application, where the dimensions of the parameters are larger than the number of observations. In this paper, we are interested in estimating the fixed effects parameters of the LMM when it is assumed that some prior information is available in the form of linear restrictions on the parameters. We propose the pretest and shrinkage estimation strategies using the ridge full model as the base estimator. We establish the asymptotic distributional bias and risks of the suggested estimators and investigate their relative performance with respect to the ridge full model estimator. Furthermore, we compare the numerical performance of the LASSO-type estimators with the pretest and shrinkage ridge estimators. The methodology is investigated using simulation studies and then demonstrated on an application exploring how effective brain connectivity in the default mode network (DMN) may be related to genetics within the context of Alzheimer’s disease.

Two-stage procedure in a first-order autoregressive process and comparison with a purely sequential procedure

A two-stage procedure in a first-order autoregressive model is considered that investigates the point and the interval estimation of parameters based on the least squares estimator. The two-stage procedure is shown to be as effective as the best fixed-sample-size procedure. In this regard, the significant properties of the procedure, such as asymptotic risk efficiency, asymptotic efficiency, and asymptotic consistency, are established. A Monte Carlo simulation study is conducted to compare the performance of the two-stage procedure and the purely sequential procedure. Finally, real-time series data are considered to illustrate the applicability of the two-stage procedure.

STRETCHING THE NET: MULTIDIMENSIONAL REGULARIZATION

This paper derives asymptotic risk (expected loss) results for shrinkage estimators with multidimensional regularization in high-dimensional settings. We introduce a class of multidimensional shrinkage estimators (MuSEs), which includes the elastic net, and show that—as the number of parameters to estimate grows—the empirical loss converges to the oracle-optimal risk. This result holds when the regularization parameters are estimated empirically via cross-validation or Stein’s unbiased risk estimate. To help guide applied researchers in their choice of estimator, we compare the empirical Bayes risk of the lasso, ridge, and elastic net in a spike and normal setting. Of the three estimators, we find that the elastic net performs best when the data are moderately sparse and the lasso performs best when the data are highly sparse. Our analysis suggests that applied researchers who are unsure about the level of sparsity in their data might benefit from using MuSEs such as the elastic net. We exploit these insights to propose a new estimator, the cubic net, and demonstrate through simulations that it outperforms the three other estimators for any sparsity level.

Robust estimation for Binomial conditionally nonlinear autoregressive time series based on multivariate conditional frequencies

A topical problem of robust statistical estimation of parameters for binomial conditionally nonlinear autoregressive (BiCNAR) time series under innovation outliers is considered. This problem is solved by means of s-order Markov properties for observed time series and probabilistic properties of multivariate conditional frequencies of the future state under its s-prehistory. The new robust statistical estimator ζˆ called frequencies-based estimator (FBE) is constructed for the BiCNAR parameters under innovation outliers with arbitrary discrete probability distribution having some fixed known expectation. Under mild regularity conditions the constructed FBE is shown to have the robustness properties: consistency and asymptotic normality with obtained asymptotic covariance matrix. FBE also has computational advantages: an explicit form and a fast recursive re-estimation algorithm for extension of the model. Asymptotic risk functional and its minimum are evaluated using Fisher information matrix for the considered model. Sensitivity analysis of the statistical estimator ζ̃ for the BiCNAR parameters, that is constructed for the hypothetical model without outliers, is carried out for the situation with innovation outliers: ζ̃ is shown to be inconsistent in this situation, its bias and the instability coefficient are evaluated and analyzed. The robust estimator ζˆ has a free parameter — weight matrix H. The optimal weight matrix H∗ is found by minimization of the asymptotic risk w.r.t. H. Statistical estimator for H∗ based on the observed time series is constructed. Results of multiple computer experiments on simulated and real data illustrate the theory.