Dimensional Linear Regression Research Articles

Testing Many Zero Restrictions in a High Dimensional Linear Regression Setting

We propose a test of many zero parameter restrictions in a high dimensional linear iid regression model with k ≫ n regressors. The test statistic is formed by estimating key parameters one at a time based on many low dimension regression models with nuisance terms. The parsimoniously parametrized models identify whether the original parameter of interest is or is not zero. Estimating fixed low dimension sub-parameters ensures greater estimator accuracy, it does not require a sparsity assumption nor therefore a regularized estimator, it is computationally fast compared to, e.g., de-biased Lasso, and using only the largest in a sequence of weighted estimators reduces test statistic complexity and therefore estimation error. We provide a parametric wild bootstrap for p-value computation, and prove the test is consistent and has non-trivial n / { ln ( n ) M n } -local-to-null power where M n is the l ∞ covariate fourth moment.

High–dimensional local linear regression under sparsity and convex losses

High–dimensional local linear regression under sparsity and convex losses

Robust adaptive variable selection in ultra-high dimensional linear regression models

We consider the problem of simultaneous variable selection and parameter estimation in an ultra-high dimensional linear regression model. The adaptive penalty functions are used in this regard to achieve the oracle variable selection property with simpler assumptions and lesser computational burden. Noting the non-robust nature of the usual adaptive procedures (e.g. adaptive LASSO) based on the squared error loss function against data contamination, quite frequent with modern large-scale data sets (e.g. noisy gene expression data, spectra and spectral data), in this paper, we present a new adaptive regularization procedure using a robust loss function based on the density power divergence (DPD) measure under a general class of error distributions. We theoretically prove that the proposed adaptive DPD-LASSO estimator of the regression coefficients is highly robust, consistent, asymptotically normal and leads to robust oracle-consistent variable selection under easily verifiable assumptions. Numerical illustrations are provided for the mostly used normal and heavy-tailed error densities. Finally, the proposal is applied to analyse an interesting spectral dataset, in the field of chemometrics, regarding the electron-probe X-ray microanalysis (EPXMA) of archaeological glass vessels from the 16th and 17th centuries.

The eigenvector LSD of information plus noise matrices and its application to linear regression model

In this paper, we investigate the limit of the eigenvector empirical spectral distribution (VESD) of large dimensional information plus noise matrices Cn=1nTn1/2(Xn+Rn)(Xn+Rn)′Tn1/2, where Xn are p×n random matrices with independent and identically distributed entries, Tn and Rn are non-random matrices. It is shown that as p/n→c∈(0,∞), the VESD of Cn has a deterministic limit under some mild conditions. The theoretical result is applied to the model selection problems in high dimensional linear regression model.

Comparison of Some Group Variable Selection Methods in High Dimensional Multiple Linear Regression via Simulation Study

In many applications, covariates possess a grouping structure that can be incorporated into the analysis to select important groups as well as important members of those groups. In this paper, we reviewed some group variable selection methods in the penalized regression model. This paper investigates by comparing the performance of seven previously proposed group variable selection methods; the group Lasso estimates, the group Lasso net estimates, the sparse group Lasso estimates, the group scad estimates, the group scad net estimates, the group mcp estimates, and the group gel estimate via a simulation study. The simulation study is used in determining which methods are best in all of the linear regression scenarios.

Kinematic coupling of the glenohumeral and scapulothoracic joints generates humeral axial rotation

Glenohumeral and scapulothoracic motion combine to generate humerothoracic motion, but their discrete contributions towards humerothoracic axial rotation have not been investigated. Understanding their contributions to axial rotation is important to judge the effects of pathology, surgical intervention, and physiotherapy. Therefore, the purpose of this study was to investigate the kinematic coupling between glenohumeral and scapulothoracic motion and determine their relative contributions towards axial rotation. Twenty healthy subjects (10 M/10F, ages 22–66) were previously recorded using biplane fluoroscopy while performing arm elevation in the coronal, scapular, and sagittal planes, and external rotation in 0° and 90° of abduction. Glenohumeral and scapulothoracic contributions towards axial rotation were computed by integrating the projection of glenohumeral and scapulothoracic angular velocity onto the humeral longitudinal axis, and analyzed using one dimensional statistical parametric mapping and linear regression. During arm elevation, scapulothoracic motion supplied 13–20° (76–94%) of axial rotation, mainly via scapulothoracic upward rotation. The contribution of scapulothoracic motion towards axial rotation was strongly correlated with glenohumeral plane of elevation during arm elevation. During external rotation, scapulothoracic motion contributed 10° (8%) towards axial rotation in 0° of abduction and 15° (15%) in 90° of abduction. The contribution of scapulothoracic motion towards humerothoracic axial rotation could explain the simultaneous changes in glenohumeral plane of elevation and axial rotation associated with some pathologies and surgeries. Understanding how humerothoracic motion results from the functional coupling of scapulothoracic and glenohumeral motions may inform diagnostic and treatment strategies by targeting the source of movement impairments in clinical populations.

Conditional Test for Ultrahigh Dimensional Linear Regression Coefficients

This paper is concerned with a conditional test for regression coefficients in ultrahigh dimensional linear models. Conditioning on a subset of important predictors in the model, we test the overall significance of regression coefficients of the remaining ultrahigh dimensional predictors. We first propose a conditional U-statistic test (CUT) based on an estimated U-statistic for a high dimensional linear regression model and prove that its null asymptotic distribution is normal under some mild assumptions. However, the empirical power of the proposed test is inversely affected by the dimensionality of predictors. To this end, we further propose a two-stage CUT with screening (CUTS) procedure to reduce the dimensionality under the sparsity assumption and enhance the empirical power based on random data splitting strategy. In the first stage, we divide data randomly into two equal halves and apply the conditional sure independence screening to the first half to reduce the dimensionality; In the second stage, we apply the proposed CUT test to the second half. To eliminate the effect of random data splitting and further enhance the empirical power, we also develop a powerful ensemble algorithm based on multiple splitting strategy and prove that the family-wise error rate is asymptotically controlled at a given significance level. We demonstrate the excellent finite-sample performances of the proposed conditional test via Monte Carlo simulations and a real data analysis.

Distributed Bayesian posterior voting strategy for massive data

<abstract><p>The emergence of massive data has driven recent interest in developing statistical learning and large-scale algorithms for analysis on distributed platforms. One of the widely used statistical approaches is split-and-conquer (SaC), which was originally performed by aggregating all local solutions through a simple average to reduce the computational burden caused by communication costs. Aiming at lower computation cost and satisfactorily acceptable accuracy, this paper extends SaC to Bayesian variable selection for ultra-high dimensional linear regression and builds BVSaC for aggregation. Suppose ultrahigh-dimensional data are stored in a distributed manner across multiple computing nodes, with each computing resource containing a disjoint subset of data. On each node machine, we perform variable selection and coefficient estimation through a hierarchical Bayes formulation. Then, a weighted majority voting method BVSaC is used to combine the local results to retain good performance. The proposed approach only requires a small portion of computation cost on each local dataset and therefore eases the computational burden, especially in Bayesian computation, meanwhile, pays a little cost to receive accuracy, which in turn increases the feasibility of analyzing extraordinarily large datasets. Simulations and a real-world example show that the proposed approach performed as well as the whole sample hierarchical Bayes method in terms of the accuracy of variable selection and estimation.</p></abstract>

A fast and efficient smoothing approach to Lasso regression and an application in statistical genetics: polygenic risk scores for chronic obstructive pulmonary disease (COPD)

High dimensional linear regression problems are often fitted using Lasso approaches. Although the Lasso objective function is convex, it is not differentiable everywhere, making the use of gradient descent methods for minimization not straightforward. To avoid this technical issue, we apply Nesterov smoothing to the original (unsmoothed) Lasso objective function. We introduce a closed-form smoothed Lasso which preserves the convexity of the Lasso function, is uniformly close to the unsmoothed Lasso, and allows us to obtain closed-form derivatives everywhere for efficient and fast minimization via gradient descent. Our simulation studies are focused on polygenic risk scores using genetic data from a genome-wide association study (GWAS) for chronic obstructive pulmonary disease (COPD). We compare accuracy and runtime of our approach to the current gold standard in the literature, the FISTA algorithm. Our results suggest that the proposed methodology provides estimates with equal or higher accuracy than the FISTA algorithm while having the same asymptotic runtime scaling. The proposed methodology is implemented in the R-package smoothedLasso, available on the Comprehensive R Archive Network (CRAN).

Sparse regression for extreme values

We study the problem of selecting features associated with extreme values in high dimensional linear regression. Normally, in linear modeling problems, the presence of abnormal extreme values or outliers is considered an anomaly which should either be removed from the data or remedied using robust regression methods. In many situations, however, the extreme values in regression modeling are not outliers but rather the signals of interest; consider traces from spiking neurons, volatility in finance, or extreme events in climate science, for example. In this paper, we propose a new method for sparse high-dimensional linear regression for extreme values which is motivated by the Subbotin, or generalized normal distribution, which we call the extreme value linear regression model. For our method, we utilize an ℓp norm loss where p is an even integer greater than two; we demonstrate that this loss increases the weight on extreme values. We prove consistency and variable selection consistency for the extreme value linear regression with a Lasso penalty, which we term the Extreme Lasso, and we also analyze the theoretical impact of extreme value observations on the model parameter estimates using the concept of influence functions. Through simulation studies and a real-world data example, we show that the Extreme Lasso outperforms other methods currently used in the literature for selecting features of interest associated with extreme values in high-dimensional regression.

Consistent Group Selection with Bayesian High Dimensional Modeling

In many applications with high dimensional covariates, the covariates are naturally structured into different groups which can be used to perform efficient statistical inference. We propose a Bayesian hierarchical model with a spike and slab prior specification to perform group selection in high dimensional linear regression models. While several penalization methods and more recently, some Bayesian approaches are proposed for group selection, theoretical properties of Bayesian approaches have not been studied extensively. In this paper, we provide novel theoretical results for group selection consistency under spike and slab priors which demonstrate that the proposed Bayesian approach has advantages compared to penalization approaches. Our theoretical results accommodate flexible conditions on the design matrix and can be applied to commonly used statistical models such as nonparametric additive models for which very limited theoretical results are available for the Bayesian methods. A shotgun stochastic search algorithm is adopted for the implementation of our proposed approach. We illustrate through simulation studies that the proposed method has better performance for group selection compared to a variety of existing methods.

Semisupervised Inference for Explained Variance in High Dimensional Linear Regression and its Applications

SummaryThe paper considers statistical inference for the explained variance βTΣβ under the high dimensional linear model Y = Xβ + ε in the semisupervised setting, where β is the regression vector and Σ is the design covariance matrix. A calibrated estimator, which efficiently integrates both labelled and unlabelled data, is proposed. It is shown that the estimator achieves the minimax optimal rate of convergence in the general semisupervised framework. The optimality result characterizes how the unlabelled data contribute to the estimation accuracy. Moreover, the limiting distribution for the proposed estimator is established and the unlabelled data have also proved useful in reducing the length of the confidence interval for the explained variance. The method proposed is extended to semisupervised inference for the unweighted quadratic functional ‖β‖22. The inference results obtained are then applied to a range of high dimensional statistical problems, including signal detection and global testing, prediction accuracy evaluation and confidence ball construction. The numerical improvement of incorporating the unlabelled data is demonstrated through simulation studies and an analysis of estimating heritability for a yeast segregant data set with multiple traits.

Grouped variable screening for ultra-high dimensional data for linear model

Ultra-high dimensional data sets often need a screening step that removes irrelevant variables prior to the main analysis. In high-dimensional linear regression, screening relevant predictors before the model estimation often yields a better prediction accuracy and much faster computation. However, most existing screening approaches target on individual predictors, thus are not able to incorporate structured predictors, such as dummy variables and grouped variables. New screening methods for naturally grouped predictors for high dimensional linear regression are presented. Two popular variable screening methods are generalized to the grouped predictors case, and also a novel screening procedure is proposed. Asymptotic sure screening properties for all three methods are established. Also empirical benefits of the screening approaches via simulation and a real data analysis are demonstrated. Specifically, a two-step analysis that does a screening followed by a sparse estimation improves the prediction accuracy as well as computing time, compared to one-stage sparse regression.

Ultrahigh dimensional precision matrix estimation via refitted cross validation

This paper develops a new estimation procedure for ultrahigh dimensional sparse precision matrix, the inverse of covariance matrix. Regularization methods have been proposed for sparse precision matrix estimation, but they may not perform well with ultrahigh dimensional data due to the spurious correlation. We propose a refitted cross validation (RCV) method for sparse precision matrix estimation based on its Cholesky decomposition, which does not require the Gaussian assumption. The proposed RCV procedure can be easily implemented with existing software for ultrahigh dimensional linear regression. We establish the consistency of the proposed RCV estimation and show that the rate of convergence of the RCV estimation without assuming banded structure is the same as that of those assuming the banded structure in Bickel and Levina (2008b). Monte Carlo studies were conducted to access the finite sample performance of the RCV estimation. Our numerical comparison shows that the RCV estimation outperforms the existing ones in various scenarios. We further apply the RCV estimation for an empirical analysis of asset allocation.

Sparse linear regression models of high dimensional covariates with non-Gaussian outliers and Berkson error-in-variable under heteroscedasticity

The use of penalized least squares based estimations in high dimensional linear regression has been received a great deal of attention during past decades and applied to variable selections under sparsity scenarios. However, it remains unclear that complex data with multiple uncertainties in measurements, such as outliers, error-in-variables and heteroscedasticity, to what extent, can influence these approaches. This paper, here, demonstrates the behaviors of least-squares based approaches in complex data problems, and suggests an alternative approach, Wald-type estimator, relying on theoretical perspectives. Numerical experiments and real data analysis are also presented to illustrate the performance of the different estimation strategies.

Alveolar ridge alterations in the maxillary anterior region after tooth extraction through orthodontic forced eruption for implant site development: a clinical CBCT study.

To investigate the effects of orthodontic forced eruption (OFE) with the straight-wire appliance in the dimensions of the alveolar process when used for extracting compromised maxillary anterior teeth and implant site development. Cone-beam computed tomography (CBCT) scans of 7 patients needing extraction of 17 maxillary anterior teeth were obtained before and immediately after OFE. Alveolar plate height and thickness measurements were performed on the buccal and palatal socket walls in CBCT sagittal cross sections. Statistical analysis included sample size calculation, paired t-test, and Wilcoxon test to evaluate alveolar plate dimensional changes and linear regression analysis to assess whether bone changes and the feasibility of implant insertion were associated to tooth type and root length, baseline alveolar plate thickness, and age. OFE caused statistically significant reduction of the buccal alveolar plate height (1.95 ± 1.83 mm) and significant increase of the palatal alveolar plate height (1.31 ± 2.41 mm) in the central tooth socket areas. Buccal reduction was associated positively to the baseline root length and negatively to the thickness of the corresponding plate in the apical level. A non-significant increase was noted in both buccal (0.23 ± 0.93 mm) and palatal (0.63 ± 1.59 mm) proximal bone. Inadequate buccal bone support hindered immediate implant placement in six sockets; however, all inserted implants showed adequate and gradually increasing stability from insertion to final restoration. OFE resulted in favourable increase in the heights of the palatal and proximal alveolar bone and significant reduction in the buccal plate height, which inhibited implant placement in 35% of the treated sockets.

Deformable registration of 3D ultrasound volumes using automatic landmark generation.

US image registration is an important task e.g. in Computer Aided Surgery. Due to tissue deformation occurring between pre-operative and interventional images often deformable registration is necessary. We present a registration method focused on surface structures (i.e. saliencies) of soft tissues like organ capsules or vessels. The main concept follows the idea of representative landmarks (so called leading points). These landmarks represent saliencies in each image in a certain region of interest. The determination of deformation was based on a geometric model assuming that saliencies could locally be described by planes. These planes were calculated from the landmarks using two dimensional linear regression. Once corresponding regions in both images were found, a displacement vector field representing the local deformation was computed. Finally, the deformed image was warped to match the pre-operative image. For error calculation we used a phantom representing the urinary bladder and the prostate. The phantom could be deformed to mimic tissue deformation. Error calculation was done using corresponding landmarks in both images. The resulting target registration error of this procedure amounted to 1.63 mm. With respect to patient data a full deformable registration was performed on two 3D-US images of the abdomen. The resulting mean distance error was 2.10 ± 0.66 mm compared to an error of 2.75 ± 0.57 mm from a simple rigid registration. A two-sided paired t-test showed a p-value < 0.001. We conclude that the method improves the results of the rigid registration considerably. Provided an appropriate choice of the filter there are many possible fields of applications.

Greedy Variance Estimation for the LASSO

Recent results have proven the minimax optimality of LASSO and related algorithms for noisy linear regression. However, these results tend to rely on variance estimators that are inefficient or optimizations that are slower than LASSO itself. We propose an efficient estimator for the noise variance in high dimensional linear regression that is faster than LASSO, only requiring p matrix–vector multiplications. We prove this estimator is consistent with a good rate of convergence, under the condition that the design matrix satisfies the restricted isometry property (RIP). In practice, our estimator scales incredibly well into high dimensions, is highly parallelizable, and only incurs a modest bias.

Comparing outcomes from competing models assessing environmental flows in the Minnesota River Basin

The ability to predictively link river hydrology to ecological effects (i.e., flow-ecology relationships) is central to understanding consequences of alternative freshwater management strategies and transparently making trade-offs. While myriad techniques for developing flow-ecology relationships exist, water managers are often concerned about not only model type, but also capacity to reliably predict ecological responses. Here, we develop flow-ecology relationships with three contrasting model development philosophies and examine consistency and robustness of model outcomes as the models are applied to six future land use scenarios in the Minnesota River Basin. All three models are developed from two data sources distributed across the basin: (1) 18 years of modeled streamflow hydrology at 1016 locations and (2) 463 fish richness monitoring events at 314 locations. Model-1 assumes hydrologic change alone is indicative of ecological response, and the relative change in seven synthetic streamflow statistics is applied as a metric of ecological impact. Model-2 develops flow-ecology relationships from a traditional, deductive modeling approach using dimensional analysis and linear regression, which assumes model adoption will increase if the tool is transparent and based on ecological and hydrological principles. Model-3 applies an inductive, machine-learning model (boosted regression trees) to develop flow-ecology relationships, which assumes that lost transparency in modeling is an acceptable trade-off for increased predictive power. All three models were applied to six “alternative futures” (i.e., land use scenarios) in the Minnesota River Basin. Hydrologic model outcomes (Model-1) show large quantitative responses to land use change and provided a useful intermediate outcome for communicating how land use change alters dimensions of streamflow. Although structurally simple, Model-2 estimated fish richness with reasonable predictive accuracy (R2 = 0.38), and dimensional analysis provided a powerful mechanism for scaling predictions across a range of watershed sizes. Although opaque in the ability to visualize outcomes, the machine learning method (Model-3) estimated fish richness with high predictive accuracy (R2 = 0.73), which may be an acceptable trade-off to managers. When applied to six land use scenarios, all three models generally predict qualitatively similar outcomes (e.g., the increased agricultural development scenario has greater ecological impacts than scenarios emphasizing biodiversity or water quality).

Regularization after retention in ultrahigh dimensional linear regression models

In ultrahigh dimensional setting, independence screening has been both theoretically and empirically proved a useful variable selection framework with low computation cost. In this work, we propose a two-step framework by using marginal information in a different perspective from independence screening. In particular, we retain significant variables rather than screening out irrelevant ones. The new method is shown to be model selection consistent in the ultrahigh dimensional linear regression model. To improve the finite sample performance, we then introduce a three-step version and characterize its asymptotic behavior. Simulations and real data analysis show advantages of our method over independence screening and its iterative variants in certain regimes.