Vector Of Regression Coefficients Research Articles

Robust multi-outcome regression with correlated covariate blocks using fused LAD-lasso

Lasso is a popular and efficient approach to simultaneous estimation and variable selection in high-dimensional regression models. In this paper, a robust fused LAD-lasso method for multiple outcomes is presented that addresses the challenges of non-normal outcome distributions and outlying observations. Measured covariate data from space or time, or spectral bands or genomic positions often have natural correlation structure arising from measuring distance between the covariates. The proposed multi-outcome approach includes handling of such covariate blocks by a group fusion penalty, which encourages similarity between neighboring regression coefficient vectors by penalizing their differences, for example, in sequential data situation. Properties of the proposed approach are illustrated by extensive simulations using BIC-type criteria for model selection. The method is also applied to a real-life skewed data on retirement behavior with longitudinal heteroscedastic explanatory variables.

Quasi-likelihood analysis for Student-Lévy regression

AbstractWe consider the quasi-likelihood analysis for a linear regression model driven by a Student-t Lévy process with constant scale and arbitrary degrees of freedom. The model is observed at high frequency over an extending period, under which we can quantify how the sampling frequency affects estimation accuracy. In that setting, joint estimation of trend, scale, and degrees of freedom is a non-trivial problem. The bottleneck is that the Student-t distribution is not closed under convolution, making it difficult to estimate all the parameters fully based on the high-frequency time scale. To efficiently deal with the intricate nature from both theoretical and computational points of view, we propose a two-step quasi-likelihood analysis: first, we make use of the Cauchy quasi-likelihood for estimating the regression-coefficient vector and the scale parameter; then, we construct the sequence of the unit-period cumulative residuals to estimate the remaining degrees of freedom. In particular, using full data in the first step causes a problem stemming from the small-time Cauchy approximation, showing the need for data thinning.

Fast Minimum Error Entropy for Linear Regression

The minimum error entropy (MEE) criterion finds extensive utility across diverse applications, particularly in contexts characterized by non-Gaussian noise. However, its computational demands are notable, and are primarily attributable to the double summation operation involved in calculating the probability density function (PDF) of the error. To address this, our study introduces a novel approach, termed the fast minimum error entropy (FMEE) algorithm, aimed at mitigating computational complexity through the utilization of polynomial expansions of the error PDF. Initially, the PDF approximation of a random variable is derived via the Gram–Charlier expansion. Subsequently, we proceed to ascertain and streamline the entropy of the random variable. Following this, the error entropy inherent to the linear regression model is delineated and expressed as a function of the regression coefficient vector. Lastly, leveraging the gradient descent algorithm, we compute the regression coefficient vector corresponding to the minimum error entropy. Theoretical scrutiny reveals that the time complexity of FMEE stands at O(n), in stark contrast to the O(n2) complexity associated with MEE. Experimentally, our findings underscore the remarkable efficiency gains afforded by FMEE, with time consumption registering less than 1‰ of that observed with MEE. Encouragingly, this efficiency leap is achieved without compromising accuracy, as evidenced by negligible differentials observed between the accuracies of FMEE and MEE. Furthermore, comprehensive regression experiments on real-world electric datasets in northwest China demonstrate that our FMEE outperforms baseline methods by a clear margin.

Sliced inverse regression via natural canonical thresholding

. In a moderate or high-dimensional framework, the sliced inverse regression (SIR) method requires the inversion of the empirical covariance matrix which yields numerical problems in estimating the central subspace. In order to improve SIR, several methods based on the regularization of the covariance matrix or the use of principal component analysis (PCA) were proposed. Yet most of these select the eigen-directions with the largest eigenvalues in an unexplained way. This article circumvents these difficulties by suggesting a new regularization of SIR based on the singular value decomposition (SVD) of the data matrix of the predictors called sliced inverse regression via natural canonical thresholding (SIR-NCT). SIR-NCT makes it possible to relate the vector of new canonical regression coefficients of the reduced dimension to the vector of the initial regression coefficients of large dimension. Moreover, we use thresholding to cut the components associated with insignificant directions. Some theoretical results are presented for SIR-SVD. Experiments with simulated and real data show that SIR-NCT outperforms its competitors.

Trending Time-Varying Coefficient Spatial Panel Data Models

This article investigates the estimation and inference of spatial panel data models in which the regression coefficient vector is a trending function. We use time differences to eliminate the individual effects and employ various GMM estimations for regression coefficients with both linear and quadratic moments. Time trend estimator based on these GMM estimations is also proposed. Monte Carlo experiments show that the finite sample performance of the estimators is satisfactory. As an empirical illustration, we investigate the trending pattern of the spillover effect of air pollution among Chinese cities from 2015 to 2021.

Multi-task learning regression via convex clustering

Multi-task learning (MTL) is a methodology that aims to improve the general performance of estimation and prediction by sharing common information among related tasks. In the MTL, there are several assumptions for the relationships and methods to incorporate them. One of the natural assumptions in the practical situation is that tasks are classified into some clusters with their characteristics. For this assumption, the group fused regularization approach performs clustering of the tasks by shrinking the difference among tasks. This enables the transfer of common information within the same cluster. However, this approach also transfers the information between different clusters, which worsens the estimation and prediction. To overcome this problem, an MTL method is proposed with a centroid parameter representing a cluster center of the task. Because this model separates parameters into the parameters for regression and the parameters for clustering, estimation and prediction accuracy for regression coefficient vectors are improved. The effectiveness of the proposed method is shown through Monte Carlo simulations and applications to real data.

Iterative Pseudo-Sparse Partial Least Square and Its Higher Order Variant: Application to Inference From High-Dimensional Biosignals

Partial Least Square (PLS) regression and its (L1 or L2 norm) regularized versions have been proposed to handle the high-dimensionality aspects of the problem at hand and select relevant features. Addressing these issues improves the generalizability of decoding the unseen data, with the severe challenge of high computational complexity. In order to avoid directly solving the L1 norm optimization problem or performing matrix inversion, this paper proposes two PLS-based algorithms, Pseudo-Sparse PLS (PS-PLS) and iterative Pseudo-Sparse Higher-Order PLS (iPS-HOPLS). In these proposed methods, we add the Pseudo-Sparsity term to reduce the L1 norm of the regression coefficient vector in a selective scheme for better importance interpretation while keeping the algorithm as simple as possible. Regarding the evaluation of the proposed methods, we investigate three critical high-dimensionality regression problems of 1) the prediction of 3D trajectory from Electrocorticography (ECoG) recordings, 2) decoding continuous fluctuation of the Electromyography (EMG) powers from recorded Magnetoencephalography (MEG) signals, and 3) continuous decoding of the finger forces from the High-Density surface Electromyogram (HD-sEMG) signals. As well as providing cognitive-relevant interpretations, the experimental results show significant improvements over the generic methods and competitive performance compared to the state-of-the-art regularized PLS approaches.

Ridge-Type Pretest and Shrinkage Estimation Strategies in Spatial Error Models with an Application to a Real Data Example

Spatial regression models are widely available across several disciplines, such as functional magnetic resonance imaging analysis, econometrics, and house price analysis. In nature, sparsity occurs when a limited number of factors strongly impact overall variation. Sparse covariance structures are common in spatial regression models. The spatial error model is a significant spatial regression model that focuses on the geographical dependence present in the error terms rather than the response variable. This study proposes an effective approach using the pretest and shrinkage ridge estimators for estimating the vector of regression coefficients in the spatial error mode, considering insignificant coefficients and multicollinearity among regressors. The study compares the performance of the proposed estimators with the maximum likelihood estimator and assesses their efficacy using real-world data and bootstrapping techniques for comparison purposes.

Shrinkage estimators of BLUE for time series regression models

The least squares estimator (LSE) seems a natural estimator of linear regression models. Whereas, if the dimension of the vector of regression coefficients is greater than 1 and the residuals are dependent, the best linear unbiased estimator (BLUE), which includes the information of the covariance matrix Γ of residual process has a better performance than LSE in the sense of mean square error. As we know the unbiased estimators are generally inadmissible, Senda and Taniguchi (2006) introduced a James–Stein type shrinkage estimator for the regression coefficients based on LSE, where the residual process is a Gaussian stationary process, and provides sufficient conditions such that the James–Stein type shrinkage estimator improves LSE. In this paper, we propose a shrinkage estimator based on BLUE. Sufficient conditions for this shrinkage estimator to improve BLUE are also given. Furthermore, since Γ is infeasible, assuming that Γ has a form of Γ=Γ(θ), we introduce a feasible version of that shrinkage estimator with replacing Γ(θ) by Γ(θˆ) which is introduced in Toyooka (1986). Additionally, we give the sufficient conditions where the feasible version improves BLUE. Besides, the results of a numerical studies confirm our approach.

Laplace Distribution Based Online Identification of Linear Systems With Robust Recursive Expectation–Maximization Algorithm

The robust online identification problem of linear systems is considered in this paper using a faster robust recursive expectation-maximization (RREM) framework. To improve the convergence rate, the outliers, which would deteriorate the identified models, are accommodated with a Laplace distribution instead of Student's t-distribution. Then, the recursive transformation of the maximum likelihood function is realized with a recursive Q-function. The extensively recognized autoregressive exogenous (ARX) models are used for the description of general linear systems. As a result, the unknown parameters, including the regression coefficient vector of the ARX models, the variance of the noise without outliers, and the scale parameter of the Laplace distribution, are determined in a recursive manner. The performance of the proposed approach is tested with a simulated continuous fermentation reactor (CFR) system example and a coupled-tank experiment.

Multiple multi-sample testing under arbitrary covariance dependency.

Modern high-throughput biomedical devices routinely produce data on a large scale, and the analysis of high-dimensional datasets has become commonplace in biomedical studies. However, given thousands or tens of thousands of measured variables in these datasets, extracting meaningful features poses a challenge. In this article, we propose a procedure to evaluate the strength of the associations between a nominal (categorical) response variable and multiple features simultaneously. Specifically, we propose a framework of large-scale multiple testing under arbitrary correlation dependency among test statistics. First, marginal multinomial regressions are performed for each feature individually. Second, we use an approach of multiple marginal models for each baseline-category pair to establish asymptotic joint normality of the stacked vector of the marginal multinomial regression coefficients. Third, we estimate the (limiting) covariance matrix between the estimated coefficients from all marginal models. Finally, our approach approximates the realized false discovery proportion of a thresholding procedure for the marginal p-values for each baseline-category logit pair. The proposed approach offers a sensible trade-off between the expected numbers of true and false findings. Furthermore, we demonstrate a practical application of the method on hyperspectral imaging data. This dataset is obtained by a matrix-assisted laser desorption/ionization (MALDI) instrument. MALDI demonstrates tremendous potential for clinical diagnosis, particularly for cancer research. In our application, the nominal response categories represent cancer (sub-)types.

Modeling Under-Dispersed Count Data by the Generalized Poisson Distribution via Two New MM Algorithms

Under-dispersed count data often appear in clinical trials, medical studies, demography, actuarial science, ecology, biology, industry and engineering. Although the generalized Poisson (GP) distribution possesses the twin properties of under- and over-dispersion, in the past 50 years, many authors only treat the GP distribution as an alternative to the negative binomial distribution for modeling over-dispersed count data. To our best knowledge, the issues of calculating maximum likelihood estimates (MLEs) of parameters in GP model without covariates and with covariates for the case of under-dispersion were not solved up to now. In this paper, we first develop a new minimization–maximization (MM) algorithm to calculate the MLEs of parameters in the GP distribution with under-dispersion, and then we develop another new MM algorithm to compute the MLEs of the vector of regression coefficients for the GP mean regression model for the case of under-dispersion. Three hypothesis tests (i.e., the likelihood ratio, Wald and score tests) are provided. Some simulations are conducted. The Bangladesh demographic and health surveys dataset is analyzed to illustrate the proposed methods and comparisons with the existing Conway–Maxwell–Poisson regression model are also presented.

Mapping Australia’s precipitation: harnessing the synergies of multi-satellite remote sensing and gauge network data

ABSTRACT Coverage, resolution, and accuracy in the spatial and temporal estimates of remotely sensed precipitation from space satellites, along with the number of instruments deployed to deliver these observations, are increasing. Of key interest in this study is the unsurpassed opportunity offered by their broad and continuous coverage to complement sparse, but more accurate, in situ rain gauge measurements for building climate resilience at the local, regional, and global scales. For many parts of the globe, this opportunity remains untapped. Australia is no exception and provides a unique challenge, given the small fraction of the continent that has rain gauges, the highly diverse climate due to its large size, and the apparent worsening of extreme weather events in both frequency and intensity. Notwithstanding this great impetus, a continent-wide record of multi-satellite-gauge fused precipitation data for Australia remains lacking. This missing data asset is a prerequisite for understanding the emerging spatiotemporal dynamics of precipitation, without which reliable forecasts would be difficult if not impossible. This study seeks to address this need. Here, we develop a method which can synergistically fuse precipitation data from different sources. More specifically, the aim of this study is to develop Precipitation Profiler-Observation Fusion and Estimation (PPrOFusE), a tool to deliver high-quality gauge and multi-satellite fused precipitation data. We test and apply this tool for Australia, but it is by no means limited in scope to this region. By design, PPrOFusE has a built-in capability to assess the strengths and weaknesses of each platform. In this case study, we fused data for a period of 22 years (2000–2022) using the rain gauge network data from the Australian Bureau of Meteorology (BOM) together with satellite data from the Japan Aerospace Exploration Agency’s (JAXA) Global Satellite Mapping of Precipitation (GSMaP) and National Oceanic and Atmospheric Administration’s (NOAA) Climate Prediction Center Morphing technique (CMORPH). Our proposed precipitation data fusion method consists of two steps. Step 1, the relationship among the three sources of data is modeled by multiple linear regression at each rain gauge location, returning the least squares estimates for the associated regression coefficient vector. Step 2, such regression coefficient vector estimates for all rain gauge locations are fitted by a spatial autoregression model, whereafter the multiple linear regression coefficient vectors for those locations void of rain gauges are predicted by spatial interpolation. Key findings are twofold. First, CMORPH is more accurate for most regions of Australia than GSMaP. Second, a clustering analysis of the fused precipitation over the last 20 years suggests two key trends on Australia’s changing climate, relative to BOM’s six major climate zones, from the previous century: (a) increased spatial variability to the north, consistent with meteorological expectations, amid a southwards expansion of the wet summer dominant zones across the continent; (b) the edge of the arid region shifts southwards and pushes out Mediterranean climate and winter dominant rainfall zones across southern Australia.

A Sharper Computational Tool for Regression

Building on previous research of Chi and Chi, this article revisits estimation in robust structured regression under the criterion. We adopt the majorization-minimization (MM) principle to design a new algorithm for updating the vector of regression coefficients. Our sharp majorization achieves faster convergence than the previous alternating proximal gradient descent algorithm by Chi and Chi. In addition, we reparameterize the model by substituting precision for scale and estimate precision via a modified Newton’s method. This simplifies and accelerates overall estimation. We also introduce distance-to-set penalties to enable constrained estimation under nonconvex constraint sets. This tactic also improves performance in coefficient estimation and structure recovery. Finally, we demonstrate the merits of our improved tactics through a rich set of simulation examples and a real data application.

Multiple two-sample testing under arbitrary covariance dependency with an application in imaging mass spectrometry.

Large-scale hypothesis testing has become a ubiquitous problem in high-dimensional statistical inference, with broad applications in various scientific disciplines. One relevant application is constituted by imaging mass spectrometry (IMS) association studies, where a large number of tests are performed simultaneously in order to identify molecular masses that are associated with a particular phenotype, for example, a cancer subtype. Mass spectra obtained from matrix-assisted laser desorption/ionization (MALDI) experiments are dependent, when considered as statistical quantities. False discovery proportion (FDP) estimation and control under arbitrary dependency structure among test statistics is an active topic in modern multiple testing research. In this context, we are concerned with the evaluation of associations between the binary outcome variable (describing the phenotype) and multiple predictors derived from MALDI measurements. We propose an inference procedure in which the correlation matrix of the test statistics is utilized. The approach is based on multiple marginal models. Specifically, we fit a marginal logistic regression model for each predictor individually. Asymptotic joint normality of the stacked vector of the marginal regression coefficients is established under standard regularity assumptions, and their (limiting) correlation matrix is estimated. The proposed method extracts common factors from the resulting empirical correlation matrix. Finally, we estimate the realized FDP of a thresholding procedure for the marginal p-values. We demonstrate a practical application of the proposed workflow to MALDI IMS data in an oncologicalcontext.

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions

In high-dimensional sparse regression, would increasing the signal-to-noise ratio while fixing the sparsity level always lead to better model selection? For high-dimensional sparse regression problems, surprisingly, in this paper we answer this question in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">negative</i> in the regime of linear sparsity for the Lasso method, relying on a new concept we term <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">effect size heterogeneity</i> . Roughly speaking, a regression coefficient vector has high effect size heterogeneity if its nonzero entries have significantly different magnitudes. From the viewpoint of this new measure, we prove that the false and true positive rates achieve the optimal trade-off <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uniformly</i> along the Lasso path when this measure is maximal in a certain sense, and the worst trade-off is achieved when it is minimal in the sense that all nonzero effect sizes are roughly equal. Moreover, we demonstrate that the first false selection occurs much earlier when effect size heterogeneity is minimal than when it is maximal. The underlying cause of these two phenomena is, metaphorically speaking, the “competition” among variables with effect sizes of the same magnitude in entering the model. Taken together, our findings suggest that effect size heterogeneity shall serve as an important complementary measure to the sparsity of regression coefficients in the analysis of high-dimensional regression problems. Our proofs use techniques from approximate message passing theory as well as a novel technique for estimating the rank of the first false variable.

DOUBLY DEBIASED LASSO: HIGH-DIMENSIONAL INFERENCE UNDER HIDDEN CONFOUNDING.

Inferring causal relationships or related associations from observational data can be invalidated by the existence of hidden confounding. We focus on a high-dimensional linear regression setting, where the measured covariates are affected by hidden confounding and propose the Doubly Debiased Lasso estimator for individual components of the regression coefficient vector. Our advocated method simultaneously corrects both the bias due to estimation of high-dimensional parameters as well as the bias caused by the hidden confounding. We establish its asymptotic normality and also prove that it is efficient in the Gauss-Markov sense. The validity of our methodology relies on a dense confounding assumption, i.e. that every confounding variable affects many covariates. The finite sample performance is illustrated with an extensive simulation study and a genomic application.

Support vector regression with penalized likelihood

This paper is concerned with the method of support vector regression (SVR) with penalized likelihood. The ε-insensitive loss function utilized in SVR is naturally incorporated into the likelihood and is combined with the penalty for the vector of regression coefficients. We include all parameters necessary to implement SVR in the proposed penalized likelihood. An efficient algorithm to obtain estimators of parameters is provided and asymptotic results for the estimators are developed. We perform Monte Carlo simulations to confirm the effectiveness of the proposed method and report the results of applying the method to real data sets.

On estimation error bounds of the Elastic Net when p ≫ n

We study the estimation property of the Elastic Net estimator in high-dimensional linear regression models where the number of parameters p is comparable to or larger than the sample size n. In such a situation, one often assumes sparsity of the true regression coefficient vector , i.e., assuming that belongs to an -ball with radius , , for some . In this paper, we provide -estimation error bounds for the Elastic Net and naive Elastic Net estimators under a unified framework for high-dimensional analysis of M-estimators proposed by Negahban et al. [A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. Adv Neural Inf Process Syst. 2009;22:1348–1356]. We show that for both cases of exact sparsity and weak sparsity, under the same conditions on the design matrix, the Elastic Net estimator achieves a slightly better error bound than the Lasso estimator by suitably choosing the tuning parameters.

Refined least squares for support recovery

Sparse linear regression is a key step in many applications such as communication, image denoising or speech recognition. In these applications, the unknown signal of interest is often modeled using linear set of noisy equations. If the vector of regression coefficients (the signal) is sparse, then its values can be identified even if the system of equations is undetermined: recovery becomes possible with fewer observations than variables. In this work, we study the problem of exact support recovery based on noisy observations and present Refined Least Squares (ReLS). Given a set of noisy measurements, our goal is to recover the support of the (unknown) sparse signal. To recover the support of the unknown signal we use an average of multiple least squares solutions, each computed based on a subset of the full set of equations. The support is estimated by identifying the most significant coefficients of the average least squares solution. We demonstrate that in a wide variety of settings our method outperforms state-of-the-art support recovery algorithms.