High-dimensional Linear Models Research Articles

Noise covariance estimation in multi-task high-dimensional linear models

Noise covariance estimation in multi-task high-dimensional linear models

Nonconvex fusion penalties for high-dimensional hierarchical categorical variables

Hierarchical categorical data is commonly encountered in social science, genetics, and other fields. The interactions between variables in hierarchical structures introduce complexity in modeling and predicting. We focus on modeling the high-dimensional linear models with hierarchical categorical variables and introduce an efficient method. The proposed method offers computational advantages when dealing with high-dimensional categorical data. In the theoretical part, we demonstrate the uniqueness of the solution and show that the proposed estimator converges the least square solution under the high probability. Additionally, we showcase the effectiveness of our method on two real-world datasets, a cancer-reg dataset and an adult dataset, and simulated datasets, where our method outperforms comparative approaches in terms of predictive accuracy, variable selection, and model complexity.

High-dimensional multi-omics measured in controlled conditions are useful for maize platform and field trait predictions.

Transcriptomics and proteomics information collected on a platform can predict additive and non-additive effects for platform traits and additive effects for field traits. The effects of climate change in the form of drought, heat stress, and irregular seasonal changes threaten global crop production. The ability of multi-omics data, such as transcripts and proteins, to reflect a plant's response to such climatic factors can be capitalized in prediction models to maximize crop improvement. Implementing multi-omics characterization in field evaluations is challenging due to high costs. It is, however, possible to do it on reference genotypes in controlled conditions. Using omics measured on a platform, we tested different multi-omics-based prediction approaches, using a high dimensional linear mixed model (MegaLMM) to predict genotypes for platform traits and agronomic field traits in a panel of 244 maize hybrids. We considered two prediction scenarios: in the first one, new hybrids are predicted (CV-NH), and in the second one, partially observed hybrids are predicted (CV-POH). For both scenarios, all hybrids were characterized for omics on the platform. We observed that omics can predict both additive and non-additive genetic effects for the platform traits, resulting in much higher predictive abilities than GBLUP. It highlights their efficiency in capturing regulatory processes in relation to growth conditions. For the field traits, we observed that the additive components of omics only slightly improved predictive abilities for predicting new hybrids (CV-NH, model MegaGAO) and for predicting partially observed hybrids (CV-POH, model GAOxW-BLUP) in comparison to GBLUP. We conclude that measuring the omics in the fields would be of considerable interest in predicting productivity if the costs of omics drop significantly.

Early stopping for L2-boosting in high-dimensional linear models

Early stopping for L2-boosting in high-dimensional linear models

Testing Many Zero Restrictions in a High Dimensional Linear Regression Setting

We propose a test of many zero parameter restrictions in a high dimensional linear iid regression model with k ≫ n regressors. The test statistic is formed by estimating key parameters one at a time based on many low dimension regression models with nuisance terms. The parsimoniously parametrized models identify whether the original parameter of interest is or is not zero. Estimating fixed low dimension sub-parameters ensures greater estimator accuracy, it does not require a sparsity assumption nor therefore a regularized estimator, it is computationally fast compared to, e.g., de-biased Lasso, and using only the largest in a sequence of weighted estimators reduces test statistic complexity and therefore estimation error. We provide a parametric wild bootstrap for p-value computation, and prove the test is consistent and has non-trivial n / { ln ( n ) M n } -local-to-null power where M n is the l ∞ covariate fourth moment.

G.ridge: An R Package for Generalized Ridge Regression for Sparse and High-Dimensional Linear Models

Ridge regression is one of the most popular shrinkage estimation methods for linear models. Ridge regression effectively estimates regression coefficients in the presence of high-dimensional regressors. Recently, a generalized ridge estimator was suggested that involved generalizing the uniform shrinkage of ridge regression to non-uniform shrinkage; this was shown to perform well in sparse and high-dimensional linear models. In this paper, we introduce our newly developed R package “g.ridge” (first version published on 7 December 2023) that implements both the ridge estimator and generalized ridge estimator. The package is equipped with generalized cross-validation for the automatic estimation of shrinkage parameters. The package also includes a convenient tool for generating a design matrix. By simulations, we test the performance of the R package under sparse and high-dimensional settings with normal and skew-normal error distributions. From the simulation results, we conclude that the generalized ridge estimator is superior to the benchmark ridge estimator based on the R package “glmnet”. Hence the generalized ridge estimator may be the most recommended estimator for sparse and high-dimensional models. We demonstrate the package using intracerebral hemorrhage data.

Reprint: Statistical inference for linear mediation models with high-dimensional mediators and application to studying stock reaction to COVID-19 pandemic

Mediation analysis draws increasing attention in many research areas such as economics, finance and social sciences. In this paper, we propose new statistical inference procedures for high dimensional mediation models, in which both the outcome model and the mediator model are linear with high dimensional mediators. Traditional procedures for mediation analysis cannot be used to make statistical inference for high dimensional linear mediation models due to high-dimensionality of the mediators. We propose an estimation procedure for the indirect effects of the models via a partially penalized least squares method, and further establish its theoretical properties. We further develop a partially penalized Wald test on the indirect effects, and prove that the proposed test has a χ2 limiting null distribution. We also propose an F-type test for direct effects and show that the proposed test asymptotically follows a χ2-distribution under null hypothesis and a noncentral χ2-distribution under local alternatives. Monte Carlo simulations are conducted to examine the finite sample performance of the proposed tests and compare their performance with existing ones. We further apply the newly proposed statistical inference procedures to study stock reaction to COVID-19 pandemic via an empirical analysis of studying the mediation effects of financial metrics that bridge company’s sector and stock return.

A Unified Framework for Change Point Detection in High-Dimensional Linear Models

A Unified Framework for Change Point Detection in High-Dimensional Linear Models

Component-wise iterative ensemble Kalman inversion for static Bayesian models with unknown measurement error covariance

The ensemble Kalman filter (EnKF) is a Monte Carlo approximation of the Kalman filter for high dimensional linear Gaussian state space models. EnKF methods have also been developed for parameter inference of static Bayesian models with a Gaussian likelihood, in a way that is analogous to likelihood tempering sequential Monte Carlo (SMC). These methods are commonly referred to as ensemble Kalman inversion (EKI). Unlike SMC, the inference from EKI is asymptotically biased if the likelihood is non-linear and/or non-Gaussian and if the priors are non-Gaussian. However, it is significantly faster to run. Currently, a large limitation of EKI methods is that the covariance of the measurement error is assumed to be fully known. We develop a new method, which we call component-wise iterative EKI (CW-IEKI), that allows elements of the covariance matrix to be inferred alongside the model parameters at negligible extra cost. This novel method is compared to SMC on a linear Gaussian example as well as four examples with non-linear dynamics (i.e. non-linear function of the model parameters). The non-linear examples include a set of population models applied to synthetic data, a model of nitrogen mineralisation in soil that is based on the Agricultural Production Systems Simulator, a model predicting seagrass decline due to stress from water temperature and light, and a model predicting coral calcification rates. On our examples, we find that CW-IEKI has relatively similar predictive performance to SMC, albeit with greater uncertainty, and it has a significantly faster run time.

Estimation of Linear Functionals in High-Dimensional Linear Models: From Sparsity to Nonsparsity

ABSTRACT High-dimensional linear models are commonly used in practice. In many applications, one is interested in linear transformations β ⊤ x of regression coefficients β ∈ R p , where x is a specific point and is not required to be identically distributed as the training data. One common approach is the plug-in technique which first estimates β , then plugs the estimator in the linear transformation for prediction. Despite its popularity, estimation of β can be difficult for high-dimensional problems. Commonly used assumptions in the literature include that the signal of coefficients β is sparse and predictors are weakly correlated. These assumptions, however, may not be easily verified, and can be violated in practice. When β is non-sparse or predictors are strongly correlated, estimation of β can be very difficult. In this article, we propose a novel pointwise estimator for linear transformations of β . This new estimator greatly relaxes the common assumptions for high-dimensional problems, and is adaptive to the degree of sparsity of β and strength of correlations among the predictors. In particular, β can be sparse or nonsparse and predictors can be strongly or weakly correlated. The proposed method is simple for implementation. Numerical and theoretical results demonstrate the competitive advantages of the proposed method for a wide range of problems. Supplementary materials for this article are available online.

Large-Scale Two-Sample Comparison of Support Sets

ABSTRACT Two-sample multiple testing has a wide range of applications. Most of the literature considers simultaneous tests of equality of parameters. The article takes a different perspective and investigates the null hypotheses that the two support sets are equal. This formulation of the testing problem is motivated by the fact that in many applications where the two parameter vectors being compared are both sparse, one might be more concerned about the detection of differential sparsity structures rather than the difference in parameter magnitudes. Focusing on this type of problem, we develop a general approach, which adapts the newly proposed symmetry data aggregation tool combined with a novel double thresholding (DT) filter. The DT filter first constructs a sequence of pairs of ranking statistics that fulfill global symmetry properties and then chooses two data-driven thresholds along the ranking to simultaneously control the False Discovery Rate (FDR) and maximize the number of rejections. Several applications of the methodology are given including high-dimensional linear models and Gaussian graphical models. We show that the proposed method is able to asymptotically control the FDR and have power guarantee under certain conditions. Numerical results confirm the effectiveness and robustness of DT in FDR control and detection ability. Supplementary materials for this article are available online.

The eigenvector LSD of information plus noise matrices and its application to linear regression model

In this paper, we investigate the limit of the eigenvector empirical spectral distribution (VESD) of large dimensional information plus noise matrices Cn=1nTn1/2(Xn+Rn)(Xn+Rn)′Tn1/2, where Xn are p×n random matrices with independent and identically distributed entries, Tn and Rn are non-random matrices. It is shown that as p/n→c∈(0,∞), the VESD of Cn has a deterministic limit under some mild conditions. The theoretical result is applied to the model selection problems in high dimensional linear regression model.

Posw: A command for the stepwise Neyman-orthogonal estimator

Inference for structural and treatment parameters while having high-dimensional covariates in the model is increasingly common. The Neyman-orthogonal (NO) estimators in Belloni, Chernozhukov, and Wei (2016, Journal of Business and Economic Statistics 34: 606–619) produce valid inferences for the parameters of interest while using generalized linear model lasso methods to select the covariates. Drukker and Liu (2022, Econometric Reviews 41: 1047–1076) extended the estimators in Belloni, Chernozhukov, and Wei (2016) by using a Bayesian information criterion stepwise method and a testing-stepwise method as the covariate selector. Drukker and Liu (2022) found a family of data-generating processes for which the NO estimator based on Bayesian information criterion stepwise produces much more reliable inferences than the lasso-based NO estimator. In this article, we describe the implementation of posw, a command for the stepwise-based NO estimator for the high-dimensional linear, logit, and Poisson models.

Reliable Genetic Correlation Estimation via Multiple Sample Splitting and Smoothing

In this paper, we aim to investigate the problem of estimating the genetic correlation between two traits. Instead of making assumptions about the distribution of effect sizes of the genetic factors, we propose the use of a high-dimensional linear model to relate a trait to genetic factors. To estimate the genetic correlation, we develop a generic strategy that combines the use of sparse penalization methods and multiple sample splitting approaches. The final estimate is determined by taking the median of the calculations, resulting in a smoothed and reliable estimate. Through simulations, we demonstrate that our proposed approach is reliable and accurate in comparison to naive plug-in methods. To further illustrate the advantages of our method, we apply it to a real-world example of a bacterial GWAS dataset, specifically to estimate the genetic correlation between antibiotic resistant traits in Streptococus pneumoniae. This application not only validates the effectiveness of our method but also highlights its potential in real-world applications.

Debiased and thresholded ridge regression for linear models with heteroskedastic and correlated errors

Abstract High-dimensional linear models with independent errors have been well-studied. However, statistical inference on a high-dimensional linear model with heteroskedastic, dependent (and possibly nonstationary) errors is still a novel topic. Under such complex assumptions, the paper at hand introduces a debiased and thresholded ridge regression estimator that is consistent, and is able to recover the model sparsity. Moreover, we derive a Gaussian approximation theorem for the estimator, and apply a dependent wild bootstrap algorithm to construct simultaneous confidence interval and hypothesis tests for linear combinations of parameters. Numerical experiments with both real and simulated data show that the proposed estimator has good finite sample performance. Of independent interest is the development of a new class of heteroscedastic, (weakly) dependent, and nonstationary random variables that can be used as a general model for regression errors.

Partial least square based approaches for high-dimensional linear mixed models

Partial least square based approaches for high-dimensional linear mixed models

Koopman-Operator-Based Robust Data-Driven Control for Wheeled Mobile Robots

The Koopman operator theory offers a way to construct explicit control-oriented high-dimensional linear dynamical models for the original nonlinear systems, solely using the input–output data of the dynamical system. The modeling accuracy of the Koopman model largely depends on the basis functions (lifting functions), dimensionality, and data quality. However, there has not been a systematic way to solve the problems mentioned above. In this article, a Koopman-operator-based robust data-driven control framework is proposed for wheeled mobile robots, via incorporating tools from control theory, to solve the problem of modeling errors of the Koopman model. By employing an extended state observer, the modeling errors of the Koopman model, including unknown external disturbances, are online estimated and compensated in the control signal in real time. Then, sliding-mode control is used to synthesize the controller. Importantly, the method of virtual control input is proposed, to cope with the model errors arising from the rotational motion of all the mobile robots. Besides, stability analysis is conducted, and the optimal dimensionality of the Koopman model is experimentally selected. Finally, experimental tests on an omnidirectional mobile robot are carried out to verify the effectiveness of the proposed control scheme, in terms of tracking performance and robustness.

Design of c-optimal experiments for high-dimensional linear models

We study randomized designs that minimize the asymptotic variance of a debiased lasso estimator when a large pool of unlabeled data is available but measuring the corresponding responses is costly. The optimal sampling distribution arises as the solution of a semidefinite program. The improvements in efficiency that result from these optimal designs are demonstrated via simulation experiments.

Statistical inference via conditional Bayesian posteriors in high-dimensional linear regression

We propose a new method under the Bayesian framework to perform valid inference for low dimensional parameters in high dimensional linear models under sparsity constraints. Our approach is to use surrogate Bayesian posteriors based on partial regression models to remove the effect of high dimensional nuisance variables. We name the final distribution we used to conduct inference “conditional Bayesian posterior” as it is a surrogate posterior constructed conditional on quasi posterior distributions of other parameters and does not admit a fully Bayesian interpretation. Unlike existing Bayesian regularization methods, our method can be used to quantify the estimation uncertainty for arbitrarily small signals and therefore does not require variable selection consistency to guarantee its validity. Theoretically, we show that the resulting Bayesian credible intervals achieve desired coverage probabilities in the frequentist sense. Methodologically, our proposed Bayesian framework can easily incorporate popular Bayesian regularization procedures such as those based on spike and slab priors and horseshoe priors to facilitate high accuracy estimation and inference. Numerically, our proposed method rectifies the uncertainty underestimation of Bayesian shrinkage approaches and has a comparable empirical performance with state-of-the-art frequentist methods based on extensive simulation studies and a real data analysis.

Solving multistage stochastic linear programming via regularized linear decision rules: An application to hydrothermal dispatch planning

The solution of multistage stochastic linear problems (MSLPs) represents a challenge for many applications. Long-term hydrothermal dispatch planning (LHDP) materializes this challenge in a real-world problem that affects electricity markets, economies, and natural resources worldwide. No closed-form solutions are available for MSLP and the definition of non-anticipative policies with high-quality out-of-sample performance is crucial. Linear decision rules (LDRs) provide an interesting simulation-based framework for finding high-quality policies for MSLP through two-stage stochastic models. In practical applications, however, the number of parameters to be estimated when using an LDR may be close to or higher than the number of scenarios of the sample average approximation problem, thereby generating an in-sample overfit and poor performances in out-of-sample simulations. In this paper, we propose a novel regularized LDR to solve MSLP based on the AdaLASSO (adaptive least absolute shrinkage and selection operator). The goal is to use the parsimony principle, as largely studied in high-dimensional linear regression models, to obtain better out-of-sample performance for a LDR applied to MSLP. Computational experiments show that overfitting is a non-negligible threat when using the classical non-regularized LDR to solve the LHDP, one of the most studied MSLP with relevant applications in industry. Our analysis highlights the following benefits of the proposed framework in comparison to the non-regularized benchmark: 1) significant reductions in the number of non-zero coefficients (model parsimony), 2) substantial cost reductions in out-of-sample evaluations, and 3) improved spot-price profiles.