Sparse Reduced-rank Regression Research Articles

Robust Parallel Pursuit for Large-Scale Association Network Learning

Sparse reduced-rank regression is an important tool to uncover the large-scale response-predictor association network, as exemplified by modern applications such as the diffusion networks, and recommendation systems. However, the association networks recovered by existing methods are either sensitive to outliers or not scalable under the big data setup. In this paper, we propose a new statistical learning method called robust parallel pursuit (ROP) for joint estimation and outlier detection in large-scale response-predictor association network analysis. The proposed method is scalable in that it transforms the original large-scale network learning problem into a set of sparse unit-rank estimations via factor analysis, thus facilitating an effective parallel pursuit algorithm. Furthermore, we provide comprehensive theoretical guarantees including consistency in parameter estimation, rank selection, and outlier detection, and we conduct an inference procedure to quantify the uncertainty of existence of outliers. Extensive simulation studies and two real-data analyses demonstrate the effectiveness and the scalability of the suggested approach. History: Accepted by Ram Ramesh, Area Editor/Data Science & Machine Learning. Funding: This work was supported by the National Key R&D Program of China [Grant 2022YFA1008000], Natural Science Foundation of China [Grants 72071187, 72091212, 71731010, and 71921001], China Postdoctoral Science Foundation [Grant 2023M733402], and Fundamental Research Funds for the Central Universities [Grants WK3470000017, WK2040000027, and WK2040000079]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0181 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0181 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Integrative sparse reduced-rank regression via orthogonal rotation for analysis of high-dimensional multi-source data

Integrative sparse reduced-rank regression via orthogonal rotation for analysis of high-dimensional multi-source data

Envelope-based sparse reduced-rank regression for multivariate linear model

Envelope models were first proposed by Cook et al. (2010) as a method to reduce estimative and predictive variations in multivariate regression. Sparse reduced-rank regression, introduced by Chen and Huang (2012), is a widely used technique that performs dimension reduction and variable selection simultaneously in multivariate regression. In this work, we combine envelope models and sparse reduced-rank regression method to propose an envelope-based sparse reduced-rank regression estimator, and then establish its consistency, asymptotic normality and oracle property in high-dimensional data. We carry out some Monte Carlo simulation studies and also analyze two datasets to demonstrate that the proposed envelope-based sparse reduced-rank regression method displays good variable selection and prediction performance.

LARGE-SCALE MULTIVARIATE SPARSE REGRESSION WITH APPLICATIONS TO UK BIOBANK.

In high-dimensional regression problems, often a relatively small subset of the features are relevant for predicting the outcome, and methods that impose sparsity on the solution are popular. When multiple correlated outcomes are available (multitask), reduced rank regression is an effective way to borrow strength and capture latent structures that underlie the data. Our proposal is motivated by the UK Biobank population-based cohort study, where we are faced with large-scale, ultrahigh-dimensional features, and have access to a large number of outcomes (phenotypes)-lifestyle measures, biomarkers, and disease outcomes. We are hence led to fit sparse reduced-rank regression models, using computational strategies that allow us to scale to problems of this size. We use a scheme that alternates between solving the sparse regression problem and solving the reduced rank decomposition. For the sparse regression component we propose a scalable iterative algorithm based on adaptive screening that leverages the sparsity assumption and enables us to focus on solving much smaller subproblems. The full solution is reconstructed and tested via an optimality condition to make sure it is a valid solution for the original problem. We further extend the method to cope with practical issues, such as the inclusion of confounding variables and imputation of missing values among the phenotypes. Experiments on both synthetic data and the UK Biobank data demonstrate the effectiveness of the method and the algorithm. We present multiSnpnet package, available at http://github.com/junyangq/multiSnpnet that works on top of PLINK2 files, which we anticipate to be a valuable tool for generating polygenic risk scores from human genetic studies.

Sparse reduced-rank regression for simultaneous rank and variable selection via manifold optimization

We consider the problem of constructing a reduced-rank regression model whose coefficient parameter is represented as a singular value decomposition with sparse singular vectors. The traditional estimation procedure for the coefficient parameter often fails when the true rank of the parameter is high. To overcome this issue, we develop an estimation algorithm with rank and variable selection via sparse regularization and manifold optimization, which enables us to obtain an accurate estimation of the coefficient parameter even if the true rank of the coefficient parameter is high. Using sparse regularization, we can also select an optimal value of the rank. We conduct Monte Carlo experiments and a real data analysis to illustrate the effectiveness of our proposed method.

Sparse Reduced-Rank Regression for Exploratory Visualisation of Paired Multivariate Data

Abstract In genomics, transcriptomics, and related biological fields (collectively known as omics), combinations of experimental techniques can yield multiple sets of features for the same set of biological replicates. One example is Patch-seq, a method combining single-cell RNA sequencing with electrophysiological recordings from the same cells. Here we present a framework based on sparse reduced-rank regression (RRR) for obtaining an interpretable visualisation of the relationship between the transcriptomic and the electrophysiological data. We use elastic net regularisation that yields sparse solutions and allows for an efficient computational implementation. Using several Patch-seq datasets, we show that sparse RRR outperforms both sparse full-rank regression and non-sparse RRR, as well as previous sparse RRR approaches, in terms of predictive performance. We introduce a bibiplot visualisation in order to display the dominant factors determining the relationship between transcriptomic and electrophysiological properties of neurons. We believe that sparse RRR can provide a valuable tool for the exploration and visualisation of paired multivariate datasets.

Expanding the Scope of Multivariate Regression Approaches in Cross-Omics Research

Recent technological advancements and developments have led to a dramatic increase in the amount of high-dimensional data and thus have increased the demand for proper and efficient multivariate regression methods. Numerous traditional multivariate approaches such as principal component analysis have been used broadly in various research areas, including investment analysis, image identification, and population genetic structure analysis. However, these common approaches have the limitations of ignoring the correlations between responses and a low variable selection efficiency. Therefore, in this article, we introduce the reduced rank regression method and its extensions, sparse reduced rank regression and subspace assisted regression with row sparsity, which hold potential to meet the above demands and thus improve the interpretability of regression models. We conducted a simulation study to evaluate their performance and compared them with several other variable selection methods. For different application scenarios, we also provide selection suggestions based on predictive ability and variable selection accuracy. Finally, to demonstrate the practical value of these methods in the field of microbiome research, we applied our chosen method to real population-level microbiome data, the results of which validated our method. Our method extensions provide valuable guidelines for future omics research, especially with respect to multivariate regression, and could pave the way for novel discoveries in microbiome and related research fields.

Feature extraction for hyperspectral mineral domain mapping: A test of conventional and innovative methods

Hyperspectral (HS) imaging holds great potential for the mapping of geological targets. Innovative acquisition modes such as drone-borne or terrestrial remote sensing open up new scales and angles of observation, which allow to analyze small-scale, vertical, or difficult-to-access outcrops. A variety of available sensors operating in different spectral ranges can provide information about the abundance and spatial location of various geologic materials. However geological outcrops are inherently uneven and spectrally heterogeneous, may be covered by dust, lichen or weathering crusts, or contain spectrally indistinct objects, which is why classifications or domain mapping approaches are often used in geoscientific and mineral exploration applications as a means to discriminate mineral associations (e.g. ore or alteration zones) based on overall variations in HS data. Feature extraction (FE) algorithms are prominently used as a preparatory step to identify the first order variations within the data and, simultaneously, reduce noise and data dimensionality. The most established FE algorithms in geosciences are, by far, Principal Component Analysis (PCA) and Minimum Noise Fraction (MNF). Major progress has been conducted in the image processing community within the last decades, yielding innovative FE methods that incorporate spatial information for smoother and more accurate classification results. In this paper, we test the applicability of conventional (PCA, MNF) and innovative FE techniques (OTVCA: Orthogonal total variation component analysis and WSRRR: Wavelet-based sparse reduced-rank regression) on three case studies from geological HS mapping campaigns, including drone-borne mineral exploration, terrestrial paleoseismic outcrop scanning and thermal HS lithological mapping. This allows us to explore the performance of different FE approaches on complex geological data with sparse or partly inaccurate validation data. For all case studies, we demonstrate advantages of innovative FE algorithms in terms of classification accuracy and geological interpretability. We promote the use of advanced image processing methods for applications in geoscience and mineral exploration as a tool to support geological mapping activities.

Sparse reduced-rank regression for multivariate varying-coefficient models

ABSTRACT Varying-coefficient regression is a popular statistical tool that models the way a certain variable modulates the effect of other predictors nonlinearly. However, a majority of the VC regression models consider univariate responses; the case of multivariate responses have received relatively lesser attention. In this paper, we propose a robust multivariate varying-coefficient model based on rank loss that models the relationships among different responses via reduced-rank regression and penalized variable selection. Some asymptotic results are also established for the proposed methods. Using synthetic data, we investigate the finite sample performance and robustness properties of the estimator. We also illustrate our methodology by application to a real dataset on periodontal disease.

A fully Bayesian approach to sparse reduced-rank multivariate regression

In the context of high-dimensional multivariate linear regression, sparse reduced-rank regression (SRRR) provides a way to handle both variable selection and low-rank estimation problems. Although there has been extensive research on SRRR, statistical inference procedures that deal with the uncertainty due to variable selection and rank reduction are still limited. To fill this research gap, we develop a fully Bayesian approach to SRRR. A major difficulty that occurs in a fully Bayesian framework is that the dimension of parameter space varies with the selected variables and the reduced-rank. Due to the varying-dimensional problems, traditional Markov chain Monte Carlo (MCMC) methods such as Gibbs sampler and Metropolis-Hastings algorithm are inapplicable in our Bayesian framework. To address this issue, we propose a new posterior computation procedure based on the Laplace approximation within the collapsed Gibbs sampler. A key feature of our fully Bayesian method is that the model uncertainty is automatically integrated out by the proposed MCMC computation. The proposed method is examined via simulation study and real data analysis.

On Cross-Validation for Sparse Reduced Rank Regression

SummaryIn high dimensional data analysis, regularization methods pursuing sparsity and/or low rank have received much attention recently. To provide a proper amount of shrinkage, it is typical to use a grid search and a model comparison criterion to find the optimal regularization parameters. However, we show that fixing the parameters across all folds may result in an inconsistency issue, and it is more appropriate to cross-validate projection–selection patterns to obtain the best coefficient estimate. Our in-sample error studies in jointly sparse and rank deficient models lead to a new class of information criteria with four scale-free forms to bypass the estimation of the noise level. By use of an identity, we propose a novel scale-free calibration to help cross-validation to achieve the minimax optimal error rate non-asymptotically. Experiments support the efficacy of the methods proposed.

Group sparse reduced rank regression for neuroimaging genetic study.

The neuroimaging genetic study usually needs to deal with high dimensionality of both brain imaging data and genetic data, so that often resulting in the issue of curse of dimensionality. In this paper, we propose a group sparse reduced rank regression model to take the relations of both the phenotypes and the genotypes for the neuroimaging genetic study. Specifically, we propose designing a graph sparsity constraint as well as a reduced rank constraint to simultaneously conduct subspace learning and feature selection. The group sparsity constraint conducts feature selection to identify genotypes highly related to neuroimaging data, while the reduced rank constraint considers the relations among neuroimaging data to conduct subspace learning in the feature selection model. Furthermore, an alternative optimization algorithm is proposed to solve the resulting objective function and is proved to achieve fast convergence. Experimental results on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset showed that the proposed method has superiority on predicting the phenotype data by the genotype data, than the alternative methods under comparison.

Machine learning shows association between genetic variability in PPARG and cerebral connectivity in preterm infants

Preterm infants show abnormal structural and functional brain development, and have a high risk of long-term neurocognitive problems. The molecular and cellular mechanisms involved are poorly understood, but novel methods now make it possible to address them by examining the relationship between common genetic variability and brain endophenotype. We addressed the hypothesis that variability in the Peroxisome Proliferator Activated Receptor (PPAR) pathway would be related to brain development. We employed machine learning in an unsupervised, unbiased, combined analysis of whole-brain diffusion tractography together with genomewide, single-nucleotide polymorphism (SNP)-based genotypes from a cohort of 272 preterm infants, using Sparse Reduced Rank Regression (sRRR) and correcting for ethnicity and age at birth and imaging. Empirical selection frequencies for SNPs associated with cerebral connectivity ranged from 0.663 to zero, with multiple highly selected SNPs mapping to genes for PPARG (six SNPs), ITGA6 (four SNPs), and FXR1 (two SNPs). SNPs in PPARG were significantly overrepresented (ranked 7-11 and 67 of 556,000 SNPs; P < 2.2 × 10-7), and were mostly in introns or regulatory regions with predicted effects including protein coding and nonsense-mediated decay. Edge-centric graph-theoretic analysis showed that highly selected white-matter tracts were consistent across the group and important for information transfer (P < 2.2 × 10-17); they most often connected to the insula (P < 6 × 10-17). These results suggest that the inhibited brain development seen in humans exposed to the stress of a premature extrauterine environment is modulated by genetic factors, and that PPARG signaling has a previously unrecognized role in cerebral development.

Cancer Subtype Discovery Based on Integrative Model of Multigenomic Data.

One major goal of large-scale cancer omics study is to understand molecular mechanisms of cancer and find new biomedical targets. To deal with the high-dimensional multidimensional cancer omics data (DNA methylation, mRNA expression, etc.), which can be used to discover new insight on identifying cancer subtypes, clustering methods are usually used to find an effective low-dimensional subspace of the original data and then cluster cancer samples in the reduced subspace. However, due to data-type diversity and big data volume, few methods can integrate these data and map them into an effective low-dimensional subspace. In this paper, we develop a dimension-reduction and data-integration method for indentifying cancer subtypes, named Scluster. First, Scluster, respectively, projects the different original data into the principal subspaces by an adaptive sparse reduced-rank regression method. Then, a fused patient-by-patient network is obtained for these subgroups through a scaled exponential similarity kernel method. Finally, candidate cancer subtypes are identified using spectral clustering method. We demonstrate the efficiency of our Scluster method using three cancers by jointly analyzing mRNA expression, miRNA expression, and DNA methylation data. The evaluation results and analyses show that Scluster is effective for predicting survival and identifies novel cancer subtypes of large-scale multi-omics data.

Asymptotic properties of adaptive group Lasso for sparse reduced rank regression

This paper studies the asymptotic properties of the penalized least squares estimator using an adaptive group Lasso penalty for the reduced rank regression. The group Lasso penalty is defined in the way that the regression coefficients corresponding to each predictor are treated as one group. It is shown that under certain regularity conditions, the estimator can achieve the minimax optimal rate of convergence. Moreover, the variable selection consistency can also be achieved, that is, the relevant predictors can be identified with probability approaching one. In the asymptotic theory, the number of response variables, the number of predictors and the rank number are allowed to grow to infinity with the sample size. Copyright © 2016 John Wiley &amp; Sons, Ltd.

Nonconvex penalized reduced rank regression and its oracle properties in high dimensions

Sparse reduced rank regression achieves dimension reduction and variable selection simultaneously. In this paper, for a class of nonconvex penalties, we give sufficient conditions that guarantee the oracle estimator is a local minimizer and stronger conditions that guarantee it is a global minimizer, with probability tending to one in an ultra-high dimensional setting. We carry out simulations to investigate the performance of the estimator. A real data set is analyzed for illustration.

Sparse reduced-rank regression with covariance estimation

Improving the predicting performance of the multiple response regression compared with separate linear regressions is a challenging question. On the one hand, it is desirable to seek model parsimony when facing a large number of parameters. On the other hand, for certain applications it is necessary to take into account the general covariance structure for the errors of the regression model. We assume a reduced-rank regression model and work with the likelihood function with general error covariance to achieve both objectives. In addition we propose to select relevant variables for reduced-rank regression by using a sparsity-inducing penalty, and to estimate the error covariance matrix simultaneously by using a similar penalty on the precision matrix. We develop a numerical algorithm to solve the penalized regression problem. In a simulation study and real data analysis, the new method is compared with two recent methods for multivariate regression and exhibits competitive performance in prediction and variable selection.

Wavelet-Based Sparse Reduced-Rank Regression for Hyperspectral Image Restoration

In this paper, a method called wavelet-based sparse reduced-rank regression (WSRRR) is proposed for hyperspectral image restoration. The method is based on minimizing a sparse regularization problem subject to an orthogonality constraint. A cyclic descent-type algorithm is derived for solving the minimization problem. For selecting the tuning parameters, we propose a method based on Stein's unbiased risk estimation. It is shown that the hyperspectral image can be restored using a few sparse components. The method is evaluated using signal-to-noise ratio and spectral angle distance for a simulated noisy data set and by classification accuracies for a real data set. Two different classifiers, namely, support vector machines and random forest, are used in this paper. The method is compared to other restoration methods, and it is shown that WSRRR outperforms them for the simulated noisy data set. It is also shown in the experiments on a real data set that WSRRR not only effectively removes noise but also maintains more fine features compared to other methods used. WSRRR also gives higher classification accuracies.

A review of multivariate analyses in imaging genetics.

Recent advances in neuroimaging technology and molecular genetics provide the unique opportunity to investigate genetic influence on the variation of brain attributes. Since the year 2000, when the initial publication on brain imaging and genetics was released, imaging genetics has been a rapidly growing research approach with increasing publications every year. Several reviews have been offered to the research community focusing on various study designs. In addition to study design, analytic tools and their proper implementation are also critical to the success of a study. In this review, we survey recent publications using data from neuroimaging and genetics, focusing on methods capturing multivariate effects accommodating the large number of variables from both imaging data and genetic data. We group the analyses of genetic or genomic data into either a priori driven or data driven approach, including gene-set enrichment analysis, multifactor dimensionality reduction, principal component analysis, independent component analysis (ICA), and clustering. For the analyses of imaging data, ICA and extensions of ICA are the most widely used multivariate methods. Given detailed reviews of multivariate analyses of imaging data available elsewhere, we provide a brief summary here that includes a recently proposed method known as independent vector analysis. Finally, we review methods focused on bridging the imaging and genetic data by establishing multivariate and multiple genotype-phenotype-associations, including sparse partial least squares, sparse canonical correlation analysis, sparse reduced rank regression and parallel ICA. These methods are designed to extract latent variables from both genetic and imaging data, which become new genotypes and phenotypes, and the links between the new genotype-phenotype pairs are maximized using different cost functions. The relationship between these methods along with their assumptions, advantages, and limitations are discussed.

Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression

In this paper, a novel multi-view facial expression recognition method is presented. Different from most of the facial expression methods that use one view of facial feature vectors in the expression recognition, we synthesize multi-view facial feature vectors and combine them to this goal. In the facial feature extraction, we use the grids with multi-scale sizes to partition each facial image into a set of sub regions and carry out the feature extraction in each sub region. To deal with the prediction of expressions, we propose a novel group sparse reduced-rank regression (GSRRR) model to describe the relationship between the multi-view facial feature vectors and the corresponding expression class label vectors. The group sparsity of GSRRR enables us to automatically select the optimal sub regions of a face that contribute most to the expression recognition. To solve the optimization problem of GSRRR, we propose an efficient algorithm using inexact augmented Lagrangian multiplier (ALM) approach. Finally, we conduct extensive experiments on both BU-3DFE and Multi-PIE facial expression databases to evaluate the recognition performance of the proposed method. The experimental results confirm better recognition performance of the proposed method compared with the state of the art methods.