High-dimensional Linear Models Research Articles

A robust and efficient change point detection method for high-dimensional linear models

In the context of linear models, a key problem of interest is to estimate the regression coefficient. Nevertheless, in certain instances, the vector of unknown coefficient parameters in a linear regression model differs from one segment to another. In this paper, when the dimension of covariates is high, a new method is proposed to examine a linear model in which the regression coefficient of two subpopulations may be different. To achieve robustness and efficiency, we introduce modal linear regression as a means of estimating the unknown coefficient parameters. Furthermore, our proposed method is capable of selecting variables and checking change points. Under certain mild assumptions, the limiting behavior of our proposed method can be established. Additionally, an estimation algorithm based on kick-one-off and SCAD approach is developed to implement in practice. For illustration, simulation studies and a real data are considered to assess the performance of our proposed method.

Sparsified simultaneous confidence intervals for high-dimensional linear models

AbstractStatistical inference of the high-dimensional regression coefficients is challenging because the uncertainty introduced by the model selection procedure is hard to account for. Currently, the inference of the model and the inference of the coefficients are separately sought. A critical question remains unsettled; that is, is it possible to embed the inference of the model into the simultaneous inference of the coefficients? If so, then how to properly design a simultaneous inference tool with desired properties? To this end, we propose a notion of simultaneous confidence intervals called the sparsified simultaneous confidence intervals (SSCI). Our intervals are sparse in the sense that some of the intervals’ upper and lower bounds are shrunken to zero (i.e., [0, 0]), indicating the unimportance of the corresponding covariates. These covariates should be excluded from the final model. The rest of the intervals, either containing zero (e.g., $$[-1,1]$$ [ - 1 , 1 ] or [0, 1]) or not containing zero (e.g., [2, 3]), indicate the plausible and significant covariates, respectively. The SSCI intuitively suggests a lower-bound model with significant covariates only and an upper-bound model with plausible and significant covariates. The proposed method can be coupled with various selection procedures, making it ideal for comparing their uncertainty. For the proposed method, we establish desirable asymptotic properties, develop intuitive graphical tools for visualization, and justify its superior performance through simulation and real data analysis.

Bayesian grouping-Gibbs sampling estimation of high-dimensional linear model with non-sparsity

In high-dimensional linear regression models, common assumptions typically entail sparsity of regression coefficients β∈Rp. However, these assumptions may not hold when the majority, if not all, of regression coefficients are non-zeros. Statistical methods designed for sparse models may lead to substantial bias in model estimation. Therefore, this article proposes a novel Bayesian Grouping-Gibbs Sampling (BGGS) method, which departs from the common sparse assumptions in high-dimensional problems. The BGGS method leverages a grouping strategy that partitions β into distinct groups, facilitating rapid sampling in high-dimensional space. The grouping number (k) can be determined using the ‘Elbow plot’, which operates efficiently and is robust against the initial value. Theoretical analysis, under some regular conditions, guarantees model selection and parameter estimation consistency, and bound for the prediction error. Furthermore, three finite simulations are conducted to assess the competitive advantages of the proposed method in terms of parameter estimation and prediction accuracy. Finally, the BGGS method is applied to a financial dataset to explore its practical utility.

Bayesian stein-type shrinkage estimators in high-dimensional linear regression models

Bayesian stein-type shrinkage estimators in high-dimensional linear regression models

A dual-penalized approach to hypothesis testing in high-dimensional linear mediation models

The field of mediation analysis, specifically high-dimensional mediation analysis, has been arousing great interest due to its applications in genetics, economics and other areas. Mediation analysis aims to investigate how exposure variables influence outcome variable via mediators, and it is categorized into direct and indirect effects based on whether the influence is mediated. A novel hypothesis testing method, called the dual-penalized method, is proposed to test direct and indirect effects. This method offers mild conditions and sound theoretical properties. Additionally, the asymptotic distributions of the proposed estimators are established to perform hypothesis testing. Results from simulation studies demonstrate that the dual-penalized method is highly effective, especially in weak signal settings. Further more, the application of this method to the childhood trauma data set reveals a new mediator with a credible basis in biological processes.

A Heteroscedasticity-Robust Overidentifying Restriction Test with High-Dimensional Covariates

This article proposes an overidentifying restriction test for high-dimensional linear instrumental variable models. The novelty of the proposed test is that it allows the number of covariates and instruments to be larger than the sample size. The test is scale-invariant and robust to heteroscedastic errors. To construct the final test statistic, we first introduce a test based on the maximum norm of multiple parameters that could be high-dimensional. The theoretical power based on the maximum norm is higher than that in the modified Cragg-Donald test, the only existing test allowing for large-dimensional covariates. Second, following the principle of power enhancement, we introduce the power-enhanced test, with an asymptotically zero component used to enhance the power to detect some extreme alternatives with many locally invalid instruments. Finally, an empirical example of the trade and economic growth nexus demonstrates the usefulness of the proposed test.

Uncertainty quantification in high-dimensional linear models incorporating graphical structures with applications to gene set analysis.

The functions of genes in networks are typically correlated due to their functional connectivity. Variable selection methods have been developed to select important genes associated with a trait while incorporating network graphical information. However, no method has been proposed to quantify the uncertainty of individual genes under such settings. In this paper, we construct confidence intervals (CIs) and provide P-values for parameters of a high-dimensional linear model incorporating graphical structures where the number of variables p diverges with the number of observations. For combining the graphical information, we propose a graph-constrained desparsified LASSO (least absolute shrinkage and selection operator) (GCDL) estimator, which reduces dramatically the influence of high correlation of predictors and enjoys the advantage of faster computation and higher accuracy compared with the desparsified LASSO. Theoretical results show that the GCDL estimator achieves asymptotic normality. The asymptotic property of the uniform convergence is established, with which an explicit expression of the uniform CI can be derived. Extensive numerical results indicate that the GCDL estimator and its (uniform) CI perform well even when predictors are highly correlated. An R package implementing the proposed method is available at https://github.com/XiaoZhangryy/gcdl.

Uncertainty quantification in epigenetic clocks via conformalized quantile regression.

DNA methylation (DNAm) is a chemical modification of DNA that can be influenced by various factors, including age, environment, and lifestyle. An epigenetic clock is a predictive tool that measures biological age based on DNAm levels. It can provide insights into an individual's biological age, which may differ from their chronological age. This difference, known as the epigenetic age acceleration, may indicate the state of one's health and risk for age-related diseases. Moreover, epigenetic clocks are used in studies of aging to assess the effectiveness of anti-aging interventions and to understand the underlying mechanisms of aging and disease. Various epigenetic clocks have been developed using samples from different populations, tissues, and cell types, typically by training high-dimensional linear regression models with an elastic net penalty. While these models can predict mean biological age with high precision, there is a lack of uncertainty quantification which is important for interpreting the precision of age estimations and for clinical decision-making. To understand the distribution of a biological age clock beyond its mean, we propose a general pipeline for training epigenetic clocks, based on an integration of high-dimensional quantile regression and conformal prediction, to effectively reveal population heterogeneity and construct prediction intervals. Our approach produces adaptive prediction intervals not only achieving nominal coverage but also accounting for the inherent variability across individuals. By using the data collected from 728 blood samples in 11 DNAm datasets from children, we find that our quantile regression-based prediction intervals are narrower than those derived from conventional mean regression-based epigenetic clocks. This observation demonstrates an improved statistical efficiency over the existing pipeline for training epigenetic clocks. In addition, the resulting intervals have a synchronized varying pattern to age acceleration, effectively revealing cellular evolutionary heterogeneity in age patterns in different developmental stages during individual childhoods and adolescent cohort. Our findings suggest that conformalized high-dimensional quantile regression can produce valid prediction intervals and uncover underlying population heterogeneity. Although our methodology focuses on the distribution of aging in children, it is applicable to a broader range of populations to improve understanding of epigenetic age beyond the mean. This inference-based toolbox could provide valuable insights for future applications of epigenetic interventions for age-related diseases.

Noise covariance estimation in multi-task high-dimensional linear models

Noise covariance estimation in multi-task high-dimensional linear models

Nonconvex fusion penalties for high-dimensional hierarchical categorical variables

Hierarchical categorical data is commonly encountered in social science, genetics, and other fields. The interactions between variables in hierarchical structures introduce complexity in modeling and predicting. We focus on modeling the high-dimensional linear models with hierarchical categorical variables and introduce an efficient method. The proposed method offers computational advantages when dealing with high-dimensional categorical data. In the theoretical part, we demonstrate the uniqueness of the solution and show that the proposed estimator converges the least square solution under the high probability. Additionally, we showcase the effectiveness of our method on two real-world datasets, a cancer-reg dataset and an adult dataset, and simulated datasets, where our method outperforms comparative approaches in terms of predictive accuracy, variable selection, and model complexity.

High-dimensional multi-omics measured in controlled conditions are useful for maize platform and field trait predictions.

Transcriptomics and proteomics information collected on a platform can predict additive and non-additive effects for platform traits and additive effects for field traits. The effects of climate change in the form of drought, heat stress, and irregular seasonal changes threaten global crop production. The ability of multi-omics data, such as transcripts and proteins, to reflect a plant's response to such climatic factors can be capitalized in prediction models to maximize crop improvement. Implementing multi-omics characterization in field evaluations is challenging due to high costs. It is, however, possible to do it on reference genotypes in controlled conditions. Using omics measured on a platform, we tested different multi-omics-based prediction approaches, using a high dimensional linear mixed model (MegaLMM) to predict genotypes for platform traits and agronomic field traits in a panel of 244 maize hybrids. We considered two prediction scenarios: in the first one, new hybrids are predicted (CV-NH), and in the second one, partially observed hybrids are predicted (CV-POH). For both scenarios, all hybrids were characterized for omics on the platform. We observed that omics can predict both additive and non-additive genetic effects for the platform traits, resulting in much higher predictive abilities than GBLUP. It highlights their efficiency in capturing regulatory processes in relation to growth conditions. For the field traits, we observed that the additive components of omics only slightly improved predictive abilities for predicting new hybrids (CV-NH, model MegaGAO) and for predicting partially observed hybrids (CV-POH, model GAOxW-BLUP) in comparison to GBLUP. We conclude that measuring the omics in the fields would be of considerable interest in predicting productivity if the costs of omics drop significantly.

Early stopping for L2-boosting in high-dimensional linear models

Early stopping for L2-boosting in high-dimensional linear models

Testing Many Zero Restrictions in a High Dimensional Linear Regression Setting

We propose a test of many zero parameter restrictions in a high dimensional linear iid regression model with k ≫ n regressors. The test statistic is formed by estimating key parameters one at a time based on many low dimension regression models with nuisance terms. The parsimoniously parametrized models identify whether the original parameter of interest is or is not zero. Estimating fixed low dimension sub-parameters ensures greater estimator accuracy, it does not require a sparsity assumption nor therefore a regularized estimator, it is computationally fast compared to, e.g., de-biased Lasso, and using only the largest in a sequence of weighted estimators reduces test statistic complexity and therefore estimation error. We provide a parametric wild bootstrap for p-value computation, and prove the test is consistent and has non-trivial n / { ln ( n ) M n } -local-to-null power where M n is the l ∞ covariate fourth moment.

G.ridge: An R Package for Generalized Ridge Regression for Sparse and High-Dimensional Linear Models

Ridge regression is one of the most popular shrinkage estimation methods for linear models. Ridge regression effectively estimates regression coefficients in the presence of high-dimensional regressors. Recently, a generalized ridge estimator was suggested that involved generalizing the uniform shrinkage of ridge regression to non-uniform shrinkage; this was shown to perform well in sparse and high-dimensional linear models. In this paper, we introduce our newly developed R package “g.ridge” (first version published on 7 December 2023) that implements both the ridge estimator and generalized ridge estimator. The package is equipped with generalized cross-validation for the automatic estimation of shrinkage parameters. The package also includes a convenient tool for generating a design matrix. By simulations, we test the performance of the R package under sparse and high-dimensional settings with normal and skew-normal error distributions. From the simulation results, we conclude that the generalized ridge estimator is superior to the benchmark ridge estimator based on the R package “glmnet”. Hence the generalized ridge estimator may be the most recommended estimator for sparse and high-dimensional models. We demonstrate the package using intracerebral hemorrhage data.

Reprint: Statistical inference for linear mediation models with high-dimensional mediators and application to studying stock reaction to COVID-19 pandemic

Mediation analysis draws increasing attention in many research areas such as economics, finance and social sciences. In this paper, we propose new statistical inference procedures for high dimensional mediation models, in which both the outcome model and the mediator model are linear with high dimensional mediators. Traditional procedures for mediation analysis cannot be used to make statistical inference for high dimensional linear mediation models due to high-dimensionality of the mediators. We propose an estimation procedure for the indirect effects of the models via a partially penalized least squares method, and further establish its theoretical properties. We further develop a partially penalized Wald test on the indirect effects, and prove that the proposed test has a χ2 limiting null distribution. We also propose an F-type test for direct effects and show that the proposed test asymptotically follows a χ2-distribution under null hypothesis and a noncentral χ2-distribution under local alternatives. Monte Carlo simulations are conducted to examine the finite sample performance of the proposed tests and compare their performance with existing ones. We further apply the newly proposed statistical inference procedures to study stock reaction to COVID-19 pandemic via an empirical analysis of studying the mediation effects of financial metrics that bridge company’s sector and stock return.

A Unified Framework for Change Point Detection in High-Dimensional Linear Models

A Unified Framework for Change Point Detection in High-Dimensional Linear Models

Component-wise iterative ensemble Kalman inversion for static Bayesian models with unknown measurement error covariance

The ensemble Kalman filter (EnKF) is a Monte Carlo approximation of the Kalman filter for high dimensional linear Gaussian state space models. EnKF methods have also been developed for parameter inference of static Bayesian models with a Gaussian likelihood, in a way that is analogous to likelihood tempering sequential Monte Carlo (SMC). These methods are commonly referred to as ensemble Kalman inversion (EKI). Unlike SMC, the inference from EKI is asymptotically biased if the likelihood is non-linear and/or non-Gaussian and if the priors are non-Gaussian. However, it is significantly faster to run. Currently, a large limitation of EKI methods is that the covariance of the measurement error is assumed to be fully known. We develop a new method, which we call component-wise iterative EKI (CW-IEKI), that allows elements of the covariance matrix to be inferred alongside the model parameters at negligible extra cost. This novel method is compared to SMC on a linear Gaussian example as well as four examples with non-linear dynamics (i.e. non-linear function of the model parameters). The non-linear examples include a set of population models applied to synthetic data, a model of nitrogen mineralisation in soil that is based on the Agricultural Production Systems Simulator, a model predicting seagrass decline due to stress from water temperature and light, and a model predicting coral calcification rates. On our examples, we find that CW-IEKI has relatively similar predictive performance to SMC, albeit with greater uncertainty, and it has a significantly faster run time.

High-dimensional variable selection accounting for heterogeneity in regression coefficients across multiple data sources.

When analyzing data combined from multiple sources (e.g., hospitals, studies), the heterogeneity across different sources must be accounted for. In this paper, we consider high-dimensional linear regression models for integrative data analysis. We propose a new adaptive clustering penalty (ACP) method to simultaneously select variables and cluster source-specific regression coefficients with sub-homogeneity. We show that the estimator based on the ACP method enjoys a strong oracle property under certain regularity conditions. We also develop an efficient algorithm based on the alternating direction method of multipliers (ADMM) for parameter estimation. We conduct simulation studies to compare the performance of the proposed method to three existing methods (a fused LASSO with adjacent fusion, a pairwise fused LASSO, and a multi-directional shrinkage penalty method). Finally, we apply the proposed method to the multi-center Childhood Adenotonsillectomy Trial to identify sub-homogeneity in the treatment effects across different study sites.

Estimation of Linear Functionals in High-Dimensional Linear Models: From Sparsity to Nonsparsity

ABSTRACT High-dimensional linear models are commonly used in practice. In many applications, one is interested in linear transformations β ⊤ x of regression coefficients β ∈ R p , where x is a specific point and is not required to be identically distributed as the training data. One common approach is the plug-in technique which first estimates β , then plugs the estimator in the linear transformation for prediction. Despite its popularity, estimation of β can be difficult for high-dimensional problems. Commonly used assumptions in the literature include that the signal of coefficients β is sparse and predictors are weakly correlated. These assumptions, however, may not be easily verified, and can be violated in practice. When β is non-sparse or predictors are strongly correlated, estimation of β can be very difficult. In this article, we propose a novel pointwise estimator for linear transformations of β . This new estimator greatly relaxes the common assumptions for high-dimensional problems, and is adaptive to the degree of sparsity of β and strength of correlations among the predictors. In particular, β can be sparse or nonsparse and predictors can be strongly or weakly correlated. The proposed method is simple for implementation. Numerical and theoretical results demonstrate the competitive advantages of the proposed method for a wide range of problems. Supplementary materials for this article are available online.

Large-Scale Two-Sample Comparison of Support Sets

ABSTRACT Two-sample multiple testing has a wide range of applications. Most of the literature considers simultaneous tests of equality of parameters. The article takes a different perspective and investigates the null hypotheses that the two support sets are equal. This formulation of the testing problem is motivated by the fact that in many applications where the two parameter vectors being compared are both sparse, one might be more concerned about the detection of differential sparsity structures rather than the difference in parameter magnitudes. Focusing on this type of problem, we develop a general approach, which adapts the newly proposed symmetry data aggregation tool combined with a novel double thresholding (DT) filter. The DT filter first constructs a sequence of pairs of ranking statistics that fulfill global symmetry properties and then chooses two data-driven thresholds along the ranking to simultaneously control the False Discovery Rate (FDR) and maximize the number of rejections. Several applications of the methodology are given including high-dimensional linear models and Gaussian graphical models. We show that the proposed method is able to asymptotically control the FDR and have power guarantee under certain conditions. Numerical results confirm the effectiveness and robustness of DT in FDR control and detection ability. Supplementary materials for this article are available online.