Principal Covariates Regression Research Articles

The PCovR biplot: a graphical tool for principal covariates regression

Biplots are useful tools because they provide a visual representation of both individuals and variables simultaneously, making it easier to explore relationships and patterns within multidimensional datasets. This paper extends their use to examine the relationship between a set of predictors X and a set of response variables Y using Principal Covariates Regression analysis (PCovR). The PCovR biplot provides a simultaneous graphical representation of individuals, predictor variables and response variables. It also provides the ability to examine the relationship between both types of variables in the form of the regression coefficient matrix.

Variable selection for both outcomes and predictors: sparse multivariate principal covariates regression

Datasets comprised of large sets of both predictor and outcome variables are becoming more widely used in research. In addition to the well-known problems of model complexity and predictor variable selection, predictive modelling with such large data also presents a relatively novel and under-studied challenge of outcome variable selection. Certain outcome variables in the data may not be adequately predicted by the given sets of predictors. In this paper, we propose the method of Sparse Multivariate Principal Covariates Regression that addresses these issues altogether by expanding the Principal Covariates Regression model to incorporate sparsity penalties on both of predictor and outcome variables. Our method is one of the first methods that perform variable selection for both predictors and outcomes simultaneously. Moreover, by relying on summary variables that explain the variance in both predictor and outcome variables, the method offers a sparse and succinct model representation of the data. In a simulation study, the method performed better than methods with similar aims such as sparse Partial Least Squares at prediction of the outcome variables and recovery of the population parameters. Lastly, we administered the method on an empirical dataset to illustrate its application in practice.

Predictive performance of psychological tests: Is it better to use items than subscales?

Using psychological tests to predict outcomes involves generating a prediction rule from these tests. For multidimensional tests, the standard approach to generate a prediction rule is to use the subscale scores of the test as predictor variables in a regression model to estimate an outcome value for each individual. The coefficients in this model are estimated with ordinary least squares and the predictive performance of the rule is estimated out-of-sample. Recently, studies used the separate items as predictors and estimated the regression coefficients with statistical learning methods to improve the predictive performance of these tests. However, it is unclear whether this approach is always beneficial. The aim is to identify factors that influence the decision whether to use items or subscales in a prediction rule, or letting the data decide between these two types of rules. Several statistical methods are used to derive the prediction rules: ordinary least squares, factor score regression, elastic net, supervised principal components, and principal covariates regression. Data from two empirical studies is analyzed and a simulation study is performed. Overall, results showed that, contrary to earlier findings, item rules are not always better than subscale rules. Subscale rules from elastic net often performed best.

Placental lesions associated with stillbirth by gestational age, according to feature importance: Results from the stillbirth collaborative research network

IntroductionPrevious studies have identified lesions commonly found in placentas associated with stillbirth but have not distinguished across a range of gestational ages (GAs). The objective of this study was to identify lesions associated with stillbirths at different GAs by adapting methods from the chemical machine learning field to assign lesion importance based on correlation with GA. MethodsPlacentas from the Stillbirth Collaborative Research Network were examined according to standard protocols. GAs at stillbirth were categorized as: <28 weeks (extreme preterm stillbirth [PTSB]), 28–33′6 weeks (early PTSB), 34–36′6 weeks (late PTSB), ≥37 weeks (term stillbirth). We identified and ranked the most discriminating placental features, as well as those that were similar across GA ranges, using Kernel Principal Covariates Regression (KPCovR). ResultsThese analyses included 210 (47.2%) extreme PTSB, 85 (19.1%) early PTSB, 62 (13.9%) late PTSB, and 88 (19.8%) term stillbirths. When we compute the KPCovR, the first principal covariate indicates that there are four lesions (acute funisitis & nucleated fetal red blood cells found in extreme PTSB; multifocal reactive amniocytes & multifocal meconium found in term stillbirth) that distinguish GA ranges among all stillbirths. DiscussionThere are distinct placental lesions present across GA ranges in stillbirths; these lesions are identifiable using sophisticated feature selection. Further investigation may identify histologic changes across gestations that relate to fetal mortality.

Logistic regression with sparse common and distinctive covariates

Having large sets of predictor variables from multiple sources concerning the same individuals is becoming increasingly common in behavioral research. On top of the variable selection problem, predicting a categorical outcome using such data gives rise to an additional challenge of identifying the processes at play underneath the predictors. These processes are of particular interest in the setting of multi-source data because they can either be associated individually with a single data source or jointly with multiple sources. Although many methods have addressed the classification problem in high dimensionality, the additional challenge of distinguishing such underlying predictor processes from multi-source data has not received sufficient attention. To this end, we propose the method of Sparse Common and Distinctive Covariates Logistic Regression (SCD-Cov-logR). The method is a multi-source extension of principal covariates regression that combines with generalized linear modeling framework to allow classification of a categorical outcome. In a simulation study, SCD-Cov-logR resulted in outperformance compared to related methods commonly used in behavioral sciences. We also demonstrate the practical usage of the method under an empirical dataset.

Clustering of variables methods and measurement models for soccer players’ performances

In sports, studying player performances is a key issue since it provides a guideline for strategic choices and helps teams in the complex procedure of buying and selling of players. In this paper we aim at investigating the ability of various composite indicators to define a measurement structure for the global soccer performance. We rely on data provided by the EA Sports experts, who are the ultimate authority on soccer performance measurement: they periodically produce a set of players’ attributes that make up the broader, theoretical performance dimensions. Considering the potential of clustering techniques to confirm or disconfirm the experts’ assumptions in terms of aggregations between indicators, 29 players’ performance attributes or variables (from the FIFA19 version of the videogame, that is, sofifa) have been considered and processed with three different techniques: the Cluster of variables around latent variables (CLV), the Principal covariates regression (PCovR) and Bayesian model-based clustering (B-MBC). The three procedures yielded clusters that differed from experts’ classification. In order to identify the most appropriate measurement structure, the resulting clusters have been embedded into Structural equation models with partial least squares (PLS-SEMs) with a Higher-Order Component (that is, the overall soccer performance). The statistically derived composite indicators have been compared with those of experts’ classification. Results support the concurrent validity of composite indicators derived through the statistical methods: overall, they show that, in the lack of expert judgement, composite indicators, as well as the resulting PLS-SEM models, are a viable alternative given their greater correlation to players’ economic value and salary.

Principal Covariates Regression for Causal Case Studies

Researcher and analyst are often interested in estimating the effect of an intervention or treatment, which takes place at the aggregate level and affect one single unit, such as country and region. Thus, comparative case studies would be their first choice in practice. However, comparative case studies could fail to yield an estimate in the effect that is unbiased and consistent, as in some contexts; there are not suitable control units that are similar to the treated. The econometric literature has taken synthetic control methods and panel data approaches to this problem. In this study, we developed a principal covariate regression estimator, which exploits the cross‐sectional correlation, as well as the temporal dependency, to reproduce the dynamics of the treated in the absence of an event or policy. From a theoretical perspective, we introduce the statistical literature on dimensional reduction to make a causal inference. From a technique perspective, we combine the vertical regression and the horizontal regression. We constructed an annual panel of 38 states, to evaluate the effect of Proposition 99 on beer sales in California, using the principal covariate regression estimator proposed here. We find that California’s tobacco control program had a significant negative and robust effect on local beer consumption, suggesting that policymakers could reduce the use of cigarette and alcohol in the public using one common behavioral intervention.

Improving sample and feature selection with principal covariates regression

Selecting the most relevant features and samples out of a large set of candidates is a task that occurs very often in the context of automated data analysis, where it improves the computational performance and often the transferability of a model. Here we focus on two popular subselection schemes applied to this end: CUR decomposition, derived from a low-rank approximation of the feature matrix, and farthest point sampling (FPS), which relies on the iterative identification of the most diverse samples and discriminating features. We modify these unsupervised approaches, incorporating a supervised component following the same spirit as the principal covariates (PCov) regression method. We show how this results in selections that perform better in supervised tasks, demonstrating with models of increasing complexity, from ridge regression to kernel ridge regression and finally feed-forward neural networks. We also present adjustments to minimise the impact of any subselection when performing unsupervised tasks. We demonstrate the significant improvements associated with PCov-CUR and PCov-FPS selections for applications to chemistry and materials science, typically reducing by a factor of two the number of features and samples required to achieve a given level of regression accuracy.

PCovR2: A flexible principal covariates regression approach to parsimoniously handle multiple criterion variables.

Principal covariates regression (PCovR) allows one to deal with the interpretational and technical problems associated with running ordinary regression using many predictor variables. In PCovR, the predictor variables are reduced to a limited number of components, and simultaneously, criterion variables are regressed on these components. By means of a weighting parameter, users can flexibly choose how much they want to emphasize reconstruction and prediction. However, when datasets contain many criterion variables, PCovR users face new interpretational problems, because many regression weights will be obtained and because some criteria might be unrelated to the predictors. We therefore propose PCovR2, which extends PCovR by also reducing the criteria to a few components. These criterion components are predicted based on the predictor components. The PCovR2 weighting parameter can again be flexibly used to focus on the reconstruction of the predictors and criteria, or on filtering out relevant predictor components and predictable criterion components. We compare PCovR2 to two other approaches, based on partial least squares (PLS) and principal components regression (PCR), that also reduce the criteria and are therefore called PLS2 and PCR2. By means of a simulated example, we show that PCovR2 outperforms PLS2 and PCR2 when one aims to recover all relevant predictor components and predictable criterion components. Moreover, we conduct a simulation study to evaluate how well PCovR2, PLS2 and PCR2 succeed in finding (1) all underlying components and (2) the subset of relevant predictor and predictable criterion components. Finally, we illustrate the use of PCovR2 by means of empirical data.

Reported chronic high glycemic diet intake predicts amyloid deposition in the precuneus of cognitively normal older adults over one year

AbstractBackgroundWe previously reported that high glycemic diet (HGD) was related to cerebral amyloid deposits, a pathologic hallmark of Alzheimer’s disease (AD), in multiple highly metabolic regions of interest (ROI). The purpose of this study was to test high glycemic diet’s relationship with longitudinal change in cerebral amyloid deposition in older adults.MethodsThis longitudinal study included 106 cognitively normal older adults with elevated (n=72) and non‐elevated (n=34) amyloid participating in a one‐year exercise intervention trial. Habitual dietary intake was collected at baseline using the National Cancer Institute’s Diet History Questionnaire (DHQ II) and intake was quantified using Diet*Calc software. We derived a HGD pattern using principal covariates regression (PCovR). Amyloid PET (Florbetapir F18) scans were obtained on a GE Discovery ST‐16PET/CT scanner at baseline and 52‐weeks. MIMNuero software was used to standardize the PET images to the whole cerebellum and calculate tracer standard uptake value ratios (SUVR) in 6 ROIs. One‐year regional SUVR delta scores were calculated. We constructed ordinary least squares regression models to assess HGD’s relationship with continuous regional delta SUVR scores controlling for age, sex, and intervention group assignment. Statistical analyses were performed using R (v. 3.6.2; R Foundation, Vienna, Austria). Statistical significance was set at p &lt; 0.05.ResultsThe participants were 68% female with a mean age of 71.1 ± 5.2 years. In the precuneus, higher intake of the derived HGD pattern (β=0.23, p=0.02), sugar (β=0.24, p=0.01), and total carbohydrate (β=0.22, p=0.03) correlated with more amyloid accumulation. Higher reported intake of carbohydrate as a percentage of total energy was related to more amyloid accumulation in the inferior medial frontal gyrus (β=0.20, p=0.04), posterior cingulate gyrus (β=0.20, p=0.04), precuneus (β=0.21, p=0.04), and superior parietal lobule (β=0.21, p=0.03).ConclusionThe results of this study corroborate our previous findings and suggest a physiological role for HGD in amyloid pathology prior to onset of AD‐related symptoms.

Structure-property maps with Kernel principal covariates regression

Data analyses based on linear methods constitute the simplest, most robust, and transparent approaches to the automatic processing of large amounts of data for building supervised or unsupervised machine learning models. Principal covariates regression (PCovR) is an underappreciated method that interpolates between principal component analysis and linear regression and can be used conveniently to reveal structure-property relations in terms of simple-to-interpret, low-dimensional maps. Here we provide a pedagogic overview of these data analysis schemes, including the use of the kernel trick to introduce an element of non-linearity while maintaining most of the convenience and the simplicity of linear approaches. We then introduce a kernelized version of PCovR and a sparsified extension, and demonstrate the performance of this approach in revealing and predicting structure-property relations in chemistry and materials science, showing a variety of examples including elemental carbon, porous silicate frameworks, organic molecules, amino acid conformers, and molecular materials.

Sparse common and distinctive covariates regression

AbstractHaving large sets of predictors from multiple sources concerning the same observation units and the same criterion is becoming increasingly common in chemometrics. When analyzing such data, chemometricians often have multiple objectives: prediction of the criterion, variable selection, and identification of underlying processes associated to individual predictor sources or to several sources jointly. Existing methods offer solutions regarding the first two aims of uncovering the predictive mechanisms and relevant variables therein for a single block of predictor variables, but the challenge of uncovering joint and distinctive predictive mechanisms and the relevant variables therein in the multisource setting still needs to be addressed. To this end, we present a multiblock extension of principal covariates regression that aims to find the complex mechanisms in which several or single sources may be involved; taken together, these mechanisms predict an outcome of interest. We call this method sparse common and distinctive covariates regression (SCD‐CovR). Through a simulation study, we demonstrate that SCD‐CovR provides competitive solutions when compared with related methods. The method is also illustrated via an application to a publicly available dataset.

Parenting Interacts with Oxytocin Polymorphisms to Predict Adolescent Social Anxiety Symptom Development: A Novel Polygenic Approach

Guided by a developmental psychopathology framework, research has increasingly focused on the interplay of genetics and environment as a predictor of different forms of psychopathology, including social anxiety. In these efforts, the polygenic nature of complex phenotypes such as social anxiety is increasingly recognized, but studies applying polygenic approaches are still scarce. In this study, we applied Principal Covariates Regression as a novel approach to creating polygenic components for the oxytocin system, which has recently been put forward as particularly relevant to social anxiety. Participants were 978 adolescents (49.4% girls; Mage T1 = 13.8 years). Across 3 years, questionnaires were used to assess adolescent social anxiety symptoms and multi-informant reports of parental psychological control and autonomy support. All adolescents were genotyped for 223 oxytocin single nucleotide polymorphisms (SNPs) in 14 genes. Using Principal Covariates Regression, these SNPs could be reduced to five polygenic components. Four components reflected the underlying linkage disequilibrium and ancestry structure, whereas the fifth component, which consisted of small contributions of many SNPs across multiple genes, was strongly positively associated with adolescent social anxiety symptoms, pointing to an index of genetic risk. Moreover, significant interactions were found with this polygenic component and the environmental variables of interest. Specifically, adolescents who scored high on this polygenic component and experienced less adequate parenting (i.e., high psychological control or low autonomy support) showed the highest levels of social anxiety. Implications of these findings are discussed in the context of individual-by-environment models.

Obtaining insights from high-dimensional data: sparse principal covariates regression

BackgroundData analysis methods are usually subdivided in two distinct classes: There are methods for prediction and there are methods for exploration. In practice, however, there often is a need to learn from the data in both ways. For example, when predicting the antibody titers a few weeks after vaccination on the basis of genomewide mRNA transcription rates, also mechanistic insights about the effect of vaccinations on the immune system are sought. Principal covariates regression (PCovR) is a method that combines both purposes. Yet, it misses insightful representations of the data as these include all the variables.ResultsHere, we propose a sparse extension of principal covariates regression such that the resulting solutions are based on an automatically selected subset of the variables. Our method is shown to outperform competing methods like sparse principal components regression and sparse partial least squares in a simulation study. Furthermore good performance of the method is illustrated on publicly available data including antibody titers and genomewide transcription rates for subjects vaccinated against the flu: the selected genes by sparse PCovR are higly enriched for immune related terms and the method predicts the titers for an independent test sample well. In comparison, no significantly enriched terms were found for the genes selected by sparse partial least squares and out-of-sample prediction was worse.ConclusionsSparse principal covariates regression is a promising and competitive tool for obtaining insights from high-dimensional data.AvailabilityThe source code implementing our proposed method is available from GitHub, together with all scripts used to extract, pre-process, analyze, and post-process the data: https://github.com/katrijnvandeun/SPCovR.

Principal Covariates Clusterwise Regression (PCCR): Accounting for Multicollinearity and Population Heterogeneity in Hierarchically Organized Data.

In the behavioral sciences, many research questions pertain to a regression problem in that one wants to predict a criterion on the basis of a number of predictors. Although in many cases, ordinary least squares regression will suffice, sometimes the prediction problem is more challenging, for three reasons: first, multiple highly collinear predictors can be available, making it difficult to grasp their mutual relations as well as their relations to the criterion. In that case, it may be very useful to reduce the predictors to a few summary variables, on which one regresses the criterion and which at the same time yields insight into the predictor structure. Second, the population under study may consist of a few unknown subgroups that are characterized by different regression models. Third, the obtained data are often hierarchically structured, with for instance, observations being nested into persons or participants within groups or countries. Although some methods have been developed that partially meet these challenges (i.e., principal covariates regression (PCovR), clusterwise regression (CR), and structural equation models), none of these methods adequately deals with all of them simultaneously. To fill this gap, we propose the principal covariates clusterwise regression (PCCR) method, which combines the key idea's behind PCovR (de Jong & Kiers in Chemom Intell Lab Syst 14(1-3):155-164, 1992) and CR (Späth in Computing 22(4):367-373, 1979). The PCCR method is validated by means of a simulation study and by applying it to cross-cultural data regarding satisfaction with life.

Model selection in principal covariates regression

Dimension-reduction based regression methods reduce the predictors to a few components and predict the criterion using these components. When applying such methods, it is often not only important to achieve good prediction of the criterion, but also desirable to gain correct information about the underlying structure of the predictors (i.e., recovery of the underlying components). In contrast to PLS and PCR, PCovR explicitly aims at achieving both goals simultaneously. Moreover, the extent to which both aspects play a role in the construction of the components can be determined flexibly with a weighting parameter. This has as a downside that a dual model selection strategy is needed: selection of the number of components and selection of the weighting parameter value. Therefore, four model selection strategies are examined, and the optimality of the extracted components is studied in comparison to those resulting from PCR and PLS analyses. Based on the results of two simulation studies, we conclude that when the questions of a researcher match the optimality criteria specified in this paper, it is advised to use PCovR rather than PCR or PLS. Moreover, we recommend to use a weighting parameter that puts a lot of emphasis on the reconstruction of the predictor scores as well as to combine the results of a scree test and a cross-validation procedure when deciding on the number of components.

PCovR: AnRPackage for Principal Covariates Regression

In this article, we present PCovR, an R package for performing principal covariates regression (PCovR; De Jong and Kiers 1992). PCovR was developed for analyzing regression data with many and/or highly collinear predictor variables. The method simultaneously reduces the predictor variables to a limited number of components and regresses the criterion variables on these components. The flexibility, interpretational advantages, and computational simplicity of PCovR make the method stand out between many other regression methods. The PCovR package offers data preprocessing options, new model selection procedures, and several component rotation strategies, some of which were not available in R up till now. The use and usefulness of the package is illustrated with a real dataset, called psychiatrists.

Regularized principal covariates regression and its application to finding coupled patterns in climate fields

There are many different methods for investigating the coupling between two climate fields, which are all based on the multivariate regression model. Each different method of solving the multivariate model has its own attractive characteristics, but often the suitability of a particular method for a particular problem is not clear. Continuum regression methods search the solution space between the conventional methods and thus can find regression model subspaces that mix the attractive characteristics of the end‐member subspaces. Principal covariates regression is a continuum regression method that is easily applied to climate fields and makes use of two end‐members: principal components regression and redundancy analysis. In this study, principal covariates regression is extended to additionally span a third end‐member (partial least squares or maximum covariance analysis). The new method, regularized principal covariates regression, has several attractive features including the following: it easily applies to problems in which the response field has missing values or is temporally sparse, it explores a wide range of model spaces, and it seeks a model subspace that will, for a set number of components, have a predictive skill that is the same or better than conventional regression methods. The new method is illustrated by applying it to the problem of predicting the southern Australian winter rainfall anomaly field using the regional atmospheric pressure anomaly field. Regularized principal covariates regression identifies four major coupled patterns in these two fields. The two leading patterns, which explain over half the variance in the rainfall field, are related to the subtropical ridge and features of the zonally asymmetric circulation.

SCREAM: A novel method for multi-way regression problems with shifts and shape changes in one mode

Some fields where calibration of multi-way data is required, such as hyphenated chromatography, can suffer of high inaccuracy when traditional N-PLS is used, due to the presence of shifts or peak shape changes in one of the modes. To overcome this problem, a new regression method for multi-way data called SCREAM (Shifted Covariates REgression Analysis for Multi-way data), which is based on a combination of PARAFAC2 and principal covariates regression (PCovR), is proposed. In particular, the algorithm combines a PARAFAC2 decomposition of the X array and a PCovR-like way of computing the regression coefficients, analogously to what has been described by Smilde and Kiers (A.K. Smilde and H.A.L. Kiers, 1999) in the case of other multi-way PCovR models. The method is tested on real and simulated datasets providing good results and performing as well or better than other available regression approaches for multi-way data.

On the selection of the weighting parameter value in Principal Covariates Regression

Ordinary linear regression falls short when many predictors are available, especially when some of these are highly correlated with (a linear combination of) other predictors. One possible solution for this problem is Principal Covariates Regression (PCovR) which combines the main ideas behind Principal Component Analysis (PCA) and regression. Like PCA, PCovR reduces the predictors to a few components and, like regression, it predicts the criterion, but using the components as predictors. The reduction of the predictors and the prediction of the criterion are conducted simultaneously, by minimizing the weighted sum of the reduction error and the prediction error. How the value of the weighting parameter α can be optimally tuned, is not so obvious however. In this paper we integrate scattered findings on this topic and derive some hypotheses on which α value is optimal in which respect and on the importance of tuning the value (how robust are the obtained results for the value that is chosen for α?). We put these hypotheses to the test by performing an extensive simulation study. As predicted, the α value that optimizes recovery of the underlying parameters and true criterion scores depends among others on the number of predictors in a specific dataset and on the ratio of the amount of error on the predictors and the amount of error on the criterion. Moreover, we show that α is mostly of influence when the components strongly differ in strength and relevance, when the number of observations almost equals the number of predictor variables, or when the criterion contains a moderate to high amount of error.