High-dimensional Statistical Modeling Research Articles

Consistent tuning parameter selection in high dimensional sparse linear regression

An exhaustive search as required for traditional variable selection methods is impractical in high dimensional statistical modeling. Thus, to conduct variable selection, various forms of penalized estimators with good statistical and computational properties, have been proposed during the past two decades. The attractive properties of these shrinkage and selection estimators, however, depend critically on the size of regularization which controls model complexity. In this paper, we consider the problem of consistent tuning parameter selection in high dimensional sparse linear regression where the dimension of the predictor vector is larger than the size of the sample. First, we propose a family of high dimensional Bayesian Information Criteria (HBIC), and then investigate the selection consistency, extending the results of the extended Bayesian Information Criterion (EBIC), in Chen and Chen (2008) to ultra-high dimensional situations. Second, we develop a two-step procedure, the SIS+AENET, to conduct variable selection in p > n situations. The consistency of tuning parameter selection is established under fairly mild technical conditions. Simulation studies are presented to confirm theoretical findings, and an empirical example is given to illustrate the use in the internet advertising data.

A Bayesian Lasso via reversible-jump MCMC

Variable selection is a topic of great importance in high-dimensional statistical modeling and has a wide range of real-world applications. Many variable selection techniques have been proposed in the context of linear regression, and the Lasso model is probably one of the most popular penalized regression techniques. In this paper, we propose a new, fully hierarchical, Bayesian version of the Lasso model by employing flexible sparsity promoting priors. To obtain the Bayesian Lasso estimate, a reversible-jump MCMC algorithm is developed for joint posterior inference over both discrete and continuous parameter spaces. Simulations demonstrate that the proposed RJ-MCMC-based Bayesian Lasso yields smaller estimation errors and more accurate sparsity pattern detection when compared with state-of-the-art optimization-based Lasso-type methods, a standard Gibbs sampler-based Bayesian Lasso and the Binomial–Gaussian prior model. To demonstrate the applicability and estimation stability of the proposed Bayesian Lasso, we examine a benchmark diabetes data set and real functional Magnetic Resonance Imaging data. As an extension of the proposed RJ-MCMC framework, we also develop an MCMC-based algorithm for the Binomial–Gaussian prior model and illustrate its improved performance over the non-Bayesian estimate via simulations.

A real survival analysis application via variable selection methods for Cox's proportional hazards model

Variable selection is fundamental to high-dimensional statistical modeling in diverse fields of sciences. In our health study, different statistical methods are applied to analyze trauma annual data, collected by 30 General Hospitals in Greece. The dataset consists of 6334 observations and 111 factors that include demographic, transport, and clinical data. The statistical methods employed in this work are the nonconcave penalized likelihood methods, Smoothly Clipped Absolute Deviation, Least Absolute Shrinkage and Selection Operator, and Hard, the maximum partial likelihood estimation method, and the best subset variable selection, adjusted to Cox's proportional hazards model and used to detect possible risk factors, which affect the length of stay in a hospital. A variety of different statistical models are considered, with respect to the combinations of factors while censored observations are present. A comparative survey reveals several differences between results and execution times of each method. Finally, we provide useful biological justification of our results.

Modeling the Genetic Etiology of Pharmacokinetic–Pharmacodynamic Links with the Arma Process

Substantial variability exists among different patients in response to drugs. The identification of genetic factors that contribute to the interpersonal differentiation has been an important task for pharmacogenetic research and drug discovery. In this article, we have derived a high-dimensional statistical model for unveiling the genetic machinery for drug response by integrating two different but biologically related processes—pharmacokinetics (PK) and pharmacodynamics (PD)—into a genetic mapping framework. Using an integrated model of PK and PD, we can identify specific DNA sequence variants and test how they relate to the differential effect of the body to the drug (PK) and the effect of the drug on the body (PD). To effectively model a two-stage hierarchic structure of the covariance matrix at the PD and PK level, we have for the first time introduced an autoregressive moving-average (ARMA) process to the mixture-based likelihood function for sequence mapping. Closed-form estimates of the determinant and inverse of the ARMA-based covariance matrix are incorporated into the estimation step, which significantly increases the computational efficiency. Simulation studies have been performed to test the statistical behavior of our model. Potential applications of this model to pharmacogenetic research are discussed.

Fixed Rank Filtering for Spatio-Temporal Data

Datasets from remote-sensing platforms and sensor networks are often spatial, temporal, and very large. Processing massive amounts of data to provide current estimates of the (hidden) state from current and past data is challenging, even for the Kalman filter. A large number of spatial locations observed through time can quickly lead to an overwhelmingly high-dimensional statistical model. Dimension reduction without sacrificing complexity is our goal in this article. We demonstrate how a Spatio-Temporal Random Effects (STRE) component of a statistical model reduces the problem to one of fixed dimension with a very fast statistical solution, a methodology we call Fixed Rank Filtering (FRF). This is compared in a simulation experiment to successive, spatial-only predictions based on an analogous Spatial Random Effects (SRE) model, and the value of incorporating temporal dependence is quantified. A remote-sensing dataset of aerosol optical depth (AOD), from the Multi-angle Imaging SpectroRadiometer (MISR) instrument on the Terra satellite, is used to compare spatio-temporal FRF with spatial-only prediction. FRF achieves rapid production of optimally filtered AOD predictions, along with their prediction standard errors. In our case, over 100,000 spatio-temporal data were processed: Parameter estimation took 64.4 seconds and optimal predictions and their standard errors took 77.3 seconds to compute. Supplemental materials giving complete details on the design and analysis of a simulation experiment, the simulation code, and the MISR data used are available on-line.

On the advantages of the non-concave penalized likelihood model selection method with minimum prediction errors in large-scale medical studies

Variable and model selection problems are fundamental to high-dimensional statistical modeling in diverse fields of sciences. Especially in health studies, many potential factors are usually introduced to determine an outcome variable. This paper deals with the problem of high-dimensional statistical modeling through the analysis of the trauma annual data in Greece for 2005. The data set is divided into the experiment and control sets and consists of 6334 observations and 112 factors that include demographic, transport and intrahospital data used to detect possible risk factors of death. In our study, different model selection techniques are applied to the experiment set and the notion of deviance is used on the control set to assess the fit of the overall selected model. The statistical methods employed in this work were the non-concave penalized likelihood methods, smoothly clipped absolute deviation, least absolute shrinkage and selection operator, and Hard, the generalized linear logistic regression, and the best subset variable selection.The way of identifying the significant variables in large medical data sets along with the performance and the pros and cons of the various statistical techniques used are discussed. The performed analysis reveals the distinct advantages of the non-concave penalized likelihood methods over the traditional model selection techniques.

Estimation and confidence sets for sparse normal mixtures

For high dimensional statistical models, researchers have begun to focus on situations which can be described as having relatively few moderately large coefficients. Such situations lead to some very subtle statistical problems. In particular, Ingster and Donoho and Jin have considered a sparse normal means testing problem, in which they described the precise demarcation or detection boundary. Meinshausen and Rice have shown that it is even possible to estimate consistently the fraction of nonzero coordinates on a subset of the detectable region, but leave unanswered the question of exactly in which parts of the detectable region consistent estimation is possible. In the present paper we develop a new approach for estimating the fraction of nonzero means for problems where the nonzero means are moderately large. We show that the detection region described by Ingster and Donoho and Jin turns out to be the region where it is possible to consistently estimate the expected fraction of nonzero coordinates. This theory is developed further and minimax rates of convergence are derived. A procedure is constructed which attains the optimal rate of convergence in this setting. Furthermore, the procedure also provides an honest lower bound for confidence intervals while minimizing the expected length of such an interval. Simulations are used to enable comparison with the work of Meinshausen and Rice, where a procedure is given but where rates of convergence have not been discussed. Extensions to more general Gaussian mixture models are also given.

Variable Selection via A Combination of the L0 and L1 Penalties

Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L1 penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L0 penalty is attractive for variable selection because it directly penalizes the number of non zero coefficients. However, the optimization involved is discontinuous and non convex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L0 and L1 penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L0 and L1 penalties in terms of variable selection while maintaining good prediction accuracy.

Variable selection using MM algorithms

Variable selection is fundamental to high-dimensional statistical modeling. Many variable selection techniques may be implemented by maximum penalized likelihood using various penalty functions. Optimizing the penalized likelihood function is often challenging because it may be nondifferentiable and/or nonconcave. This article proposes a new class of algorithms for finding a maximizer of the penalized likelihood for a broad class of penalty functions. These algorithms operate by perturbing the penalty function slightly to render it differentiable, then optimizing this differentiable function using a minorize-maximize (MM) algorithm. MM algorithms are useful extensions of the well-known class of EM algorithms, a fact that allows us to analyze the local and global convergence of the proposed algorithm using some of the techniques employed for EM algorithms. In particular, we prove that when our MM algorithms converge, they must converge to a desirable point; we also discuss conditions under which this convergence may be guaranteed. We exploit the Newton-Raphson-like aspect of these algorithms to propose a sandwich estimator for the standard errors of the estimators. Our method performs well in numerical tests.

Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

Variable selection is fundamental to high-dimensional statistical modeling, including nonparametric regression. Many approaches in use are stepwise selection procedures, which can be computationally expensive and ignore stochastic errors in the variable selection process. In this article, penalized likelihood approaches are proposed to handle these kinds of problems. The proposed methods select variables and estimate coefficients simultaneously. Hence they enable us to construct confidence intervals for estimated parameters. The proposed approaches are distinguished from others in that the penalty functions are symmetric, nonconcave on (0, ∞), and have singularities at the origin to produce sparse solutions. Furthermore, the penalty functions should be bounded by a constant to reduce bias and satisfy certain conditions to yield continuous solutions. A new algorithm is proposed for optimizing penalized likelihood functions. The proposed ideas are widely applicable. They are readily applied to a variety of parametric models such as generalized linear models and robust regression models. They can also be applied easily to nonparametric modeling by using wavelets and splines. Rates of convergence of the proposed penalized likelihood estimators are established. Furthermore, with proper choice of regularization parameters, we show that the proposed estimators perform as well as the oracle procedure in variable selection; namely, they work as well as if the correct submodel were known. Our simulation shows that the newly proposed methods compare favorably with other variable selection techniques. Furthermore, the standard error formulas are tested to be accurate enough for practical applications.

On MCMC sampling in hierarchical longitudinal models

Markov chain Monte Carlo (MCMC) algorithms have revolutionized Bayesian practice. In their simplest form (i.e., when parameters are updated one at a time) they are, however, often slow to converge when applied to high-dimensional statistical models. A remedy for this problem is to block the parameters into groups, which are then updated simultaneously using either a Gibbs or Metropolis-Hastings step. In this paper we construct several (partially and fully blocked) MCMC algorithms for minimizing the autocorrelation in MCMC samples arising from important classes of longitudinal data models. We exploit an identity used by Chib (1995) in the context of Bayes factor computation to show how the parameters in a general linear mixed model may be updated in a single block, improving convergence and producing essentially independent draws from the posterior of the parameters of interest. We also investigate the value of blocking in non-Gaussian mixed models, as well as in a class of binary response data longitudinal models. We illustrate the approaches in detail with three real-data examples.

XploRe: An Interactive Statistical Computing Environment

This book describes an interactive statistical computing environment called 1 XploRe. As the name suggests, support for exploratory statistical analysis is given by a variety of computational tools. XploRe is a matrix-oriented statistical language with a comprehensive set of basic statistical operations that provides highly interactive graphics, as well as a programming environ­ ment for user-written macros; it offers hard-wired smoothing procedures for effective high-dimensional data analysis. Its highly dynamic graphic capa­ bilities make it possible to construct student-level front ends for teaching basic elements of statistics. Hot keys make it an easy-to-use computing environment for statistical analysis. The primary objective of this book is to show how the XploRe system can be used as an effective computing environment for a large number of statistical tasks. The computing tasks we consider range from basic data matrix manipulations to interactive customizing of graphs and dynamic fit­ ting of high-dimensional statistical models. The XploRe language is similar to other statistical languages and offers an interactive help system that can be extended to user-written algorithms. The language is intuitive and read­ ers with access to other systems can, without major difficulty, reproduce the examples presented here and use them as a basis for further investigation.

The stringy lattice models

The authors propose a stringy lattice model as a natural generalization of the usual lattice model, which is parallel to a generalization from point particle theory to string theory in particle physics. They examine several examples, one of which describes a non ideal gas of strings, which changes to string liquid at some boiling temperature. As a most important example they study the light cone string field theory. They make a lattice version of it and study its properties as a statistical model. It can be regarded as a very high-dimensional statistical model in which mean-field approximation is powerful. The long string sector is shown to be dynamically trivial and the low curvature strings are statistically favoured.