Traditional Pearson Correlation Research Articles

Distance Correlation Market Graph: The Case of S&P500 Stocks

This study investigates the use of a novel market graph model for equity markets. Our graph model is built on distance correlation instead of the traditional Pearson correlation. We apply it to the study of S&P500 stocks from January 2015 to December 2022. We also compare our market graphs to the traditional market graphs in the literature, those built using Pearson correlation. To further the comparison, we also build graphs using Spearman rank correlation. Our comparisons reveal that non-linear relationships in stock returns are not captured by either Pearson correlation or Spearman rank correlation. We observe that distance correlation is a robust measure for detecting complex relationships in S&P500 stock returns. Networks built on distance correlation networks, are shown to be more responsive to market conditions during turbulent periods such as the COVID crash period.

Filling Missing Value Method for Power Quality Data Based on Correlation Analysis

A large number of monitoring indicators own strong correlation among them which help to better fill missing values in these sensor data. In this study, we propose an electric power quality data filling method based on correlation analysis. Firstly, normalized mutual information method is applied to deal with nonlinear correlation which makes up for the deficiency that the traditional Pearson correlation coefficient. Additionally, the measurement of correlation is calculated to obtain the closely correlated indicators. This study utilizes the regression model to build the strong regression model. Experimental results show that the approach can effectively improve the accuracy of filling, reduce the filling error, and improve the quality of data.

Improving Prediction Performance of Dynamic Neighbor Selection in User-Based Collaborative Filtering

Recommender systems have become more and more popular in online environments in recent years. Although different approaches are introduced to build a powerful recommender system, collaborative filtering is one of the most used approaches in the recommender systems. Yet, researchers still introduce new methods to improve prediction performances in collaborative filtering. k nearest neighbor algorithm is one of the most dominant and prevalent one in collaborative filtering. The underlying approach behind it is to select a predefined k neighbors for an active user among all users. In the traditional algorithm, the value of k is constant and is determined before the prediction process. Recently, scholars proposed to use dynamic k neighbor selection for each user. Inspired from this work, we propose to improve prediction performance, accuracy and coverage, of collaborative filtering systems under k nearest neighbor approach. We first propose that users who rate the target item should become nominees for dynamic k neighbor selection instead of all possible users whose similarities can be calculated. The similarity calculation is the most crucial point of the k nearest neighbor algorithm. Furthermore, we also propose to use the significance-weighting approach in addition to the traditional Pearson correlation coefficient when identifying the best dynamic k neighbors for each user. The experimental results on the two well-known datasets show that the prediction accuracy and coverage improve in the dynamic k neighbor selection method by selecting neighbors among users who rated the target item and introducing the significance-weighting factor into the neighbor selection phase to find more eligible neighbors.

A Brain Network Constructed on an L1-Norm Regression Model Is More Sensitive in Detecting Small World Network Changes in Early AD.

Most previous imaging studies have used traditional Pearson correlation analysis to construct brain networks. This approach fails to adequately and completely account for the interaction between adjacent brain regions. In this study, we used the L1-norm linear regression model to test the small-world attributes of the brain networks of three groups of patients, namely, those with mild cognitive impairment (MCI), Alzheimer's disease (AD), and healthy controls (HCs); we attempted to identify the method that may detect minor differences in MCI and AD patients. Twenty-four AD patients, 33 MCI patients, and 27 HC elderly subjects were subjected to functional MRI (fMRI). We applied traditional Pearson correlation and the L1-norm to construct the brain networks and then tested the small-world attributes by calculating the following parameters: clustering coefficient (Cp), path length (Lp), global efficiency (Eg), and local efficiency (Eloc). As expected, L1 could detect slight changes, mainly in MCI patients expressing higher Cp and Eloc; however, no statistical differences were found between MCI patients and HCs in terms of Cp, Lp, Eg, and Eloc, using Pearson correlation. Compared with HCs, AD patients expressed a lower Cp, Eloc, and Lp and an increased Eg using both connectivity metrics. The statistical differences between the groups indicated the brain networks constructed by the L1-norm were more sensitive to detect slight small-world network changes in early stages of AD.

Quantifying Total Influence between Variables with Information Theoretic and Machine Learning Techniques.

The increasingly sophisticated investigations of complex systems require more robust estimates of the correlations between the measured quantities. The traditional Pearson correlation coefficient is easy to calculate but sensitive only to linear correlations. The total influence between quantities is, therefore, often expressed in terms of the mutual information, which also takes into account the nonlinear effects but is not normalized. To compare data from different experiments, the information quality ratio is, therefore, in many cases, of easier interpretation. On the other hand, both mutual information and information quality ratio are always positive and, therefore, cannot provide information about the sign of the influence between quantities. Moreover, they require an accurate determination of the probability distribution functions of the variables involved. As the quality and amount of data available are not always sufficient to grant an accurate estimation of the probability distribution functions, it has been investigated whether neural computational tools can help and complement the aforementioned indicators. Specific encoders and autoencoders have been developed for the task of determining the total correlation between quantities related by a functional dependence, including information about the sign of their mutual influence. Both their accuracy and computational efficiencies have been addressed in detail, with extensive numerical tests using synthetic data. A careful analysis of the robustness against noise has also been performed. The neural computational tools typically outperform the traditional indicators in practically every respect.

Studies in the use of data mining, prediction algorithms, and a universal exchange and inference language in the analysis of socioeconomic health data

While clinical and biomedical information in digital form has been escalating, it is socioeconomic factors that are important determinants of health on the national and global scale. We show how collective use of data mining and prediction algorithms to analyze socioeconomic population health data can stand beside classical correlation analysis in routine data analysis. The underlying theoretical basis is the Dirac notation and algebra that is a scientific standard but unusual outside of the physical sciences, combined with a theory of expected information first developed for analyzing sparse data but still largely confined to bioinformatics. The latter was important here because the records analyzed (which are for US counties and equivalents, not patients) are very few by contemporary data mining standards. The approach is very unlikely to be familiar to socioeconomic researchers, so the theory and the advantages of our inference nets over the Bayes Net are reviewed here, mostly using socioeconomic examples. While our expertise and focus is in regard to novel analytical methods rather than socioeconomics per se, a significant negative (countertrending) relationship between population health and equity was initially surprising, at least to the present authors. This encouraged deeper exploration including that of the relationship between our data mining methods and traditional Pearson's correlation. The latter is susceptible to giving wrong conclusions if a phenomenon called Simpson's paradox applies, so this is also investigated. Also discussed is that, even for very few records, associative data mining can still demand significant computational resources due to a combinatorial explosion.

Bayesian Correlation Analysis for Sequence Count Data.

Evaluating the similarity of different measured variables is a fundamental task of statistics, and a key part of many bioinformatics algorithms. Here we propose a Bayesian scheme for estimating the correlation between different entities’ measurements based on high-throughput sequencing data. These entities could be different genes or miRNAs whose expression is measured by RNA-seq, different transcription factors or histone marks whose expression is measured by ChIP-seq, or even combinations of different types of entities. Our Bayesian formulation accounts for both measured signal levels and uncertainty in those levels, due to varying sequencing depth in different experiments and to varying absolute levels of individual entities, both of which affect the precision of the measurements. In comparison with a traditional Pearson correlation analysis, we show that our Bayesian correlation analysis retains high correlations when measurement confidence is high, but suppresses correlations when measurement confidence is low—especially for entities with low signal levels. In addition, we consider the influence of priors on the Bayesian correlation estimate. Perhaps surprisingly, we show that naive, uniform priors on entities’ signal levels can lead to highly biased correlation estimates, particularly when different experiments have widely varying sequencing depths. However, we propose two alternative priors that provably mitigate this problem. We also prove that, like traditional Pearson correlation, our Bayesian correlation calculation constitutes a kernel in the machine learning sense, and thus can be used as a similarity measure in any kernel-based machine learning algorithm. We demonstrate our approach on two RNA-seq datasets and one miRNA-seq dataset.

BMRF-MI: integrative identification of protein interaction network by modeling the gene dependency.

BackgroundIdentification of protein interaction network is a very important step for understanding the molecular mechanisms in cancer. Several methods have been developed to integrate protein-protein interaction (PPI) data with gene expression data for network identification. However, they often fail to model the dependency between genes in the network, which makes many important genes, especially the upstream genes, unidentified. It is necessary to develop a method to improve the network identification performance by incorporating the dependency between genes.ResultsWe proposed an approach for identifying protein interaction network by incorporating mutual information (MI) into a Markov random field (MRF) based framework to model the dependency between genes. MI is widely used in information theory to measure the uncertainty between random variables. Different from traditional Pearson correlation test, MI is capable of capturing both linear and non-linear relationship between random variables. Among all the existing MI estimators, we choose to use k-nearest neighbor MI (kNN-MI) estimator which is proved to have minimum bias. The estimated MI is integrated with an MRF framework to model the gene dependency in the context of network. The maximum a posterior (MAP) estimation is applied on the MRF-based model to estimate the network score. In order to reduce the computational complexity of finding the optimal network, a probabilistic searching algorithm is implemented. We further increase the robustness and reproducibility of the results by applying a non-parametric bootstrapping method to measure the confidence level of the identified genes. To evaluate the performance of the proposed method, we test the method on simulation data under different conditions. The experimental results show an improved accuracy in terms of subnetwork identification compared to existing methods. Furthermore, we applied our method onto real breast cancer patient data; the identified protein interaction network shows a close association with the recurrence of breast cancer, which is supported by functional annotation. We also show that the identified subnetworks can be used to predict the recurrence status of cancer patients by survival analysis.ConclusionsWe have developed an integrated approach for protein interaction network identification, which combines Markov random field framework and mutual information to model the gene dependency in PPI network. Improvements in subnetwork identification have been demonstrated with simulation datasets compared to existing methods. We then apply our method onto breast cancer patient data to identify recurrence related subnetworks. The experiment results show that the identified genes are enriched in the pathway and functional categories relevant to progression and recurrence of breast cancer. Finally, the survival analysis based on identified subnetworks achieves a good result of classifying the recurrence status of cancer patients.

CCLasso: correlation inference for compositional data through Lasso

Direct analysis of microbial communities in the environment and human body has become more convenient and reliable owing to the advancements of high-throughput sequencing techniques for 16S rRNA gene profiling. Inferring the correlation relationship among members of microbial communities is of fundamental importance for genomic survey study. Traditional Pearson correlation analysis treating the observed data as absolute abundances of the microbes may lead to spurious results because the data only represent relative abundances. Special care and appropriate methods are required prior to correlation analysis for these compositional data. In this article, we first discuss the correlation definition of latent variables for compositional data. We then propose a novel method called CCLasso based on least squares with [Formula: see text] penalty to infer the correlation network for latent variables of compositional data from metagenomic data. An effective alternating direction algorithm from augmented Lagrangian method is used to solve the optimization problem. The simulation results show that CCLasso outperforms existing methods, e.g. SparCC, in edge recovery for compositional data. It also compares well with SparCC in estimating correlation network of microbe species from the Human Microbiome Project. CCLasso is open source and freely available from https://github.com/huayingfang/CCLasso under GNU LGPL v3. dengmh@pku.edu.cn Supplementary data are available at Bioinformatics online.

Application Potential of Four Nontraditional Similarity Metrics in Hydrometeorology

AbstractThis paper presents a review and assessment of four nontraditional similarity metrics that can be applied to hydrological and meteorological data. These metrics are 1) the uncentered correlation coefficient, 2) the Hodgkin–Richards index, 3) the Petke index, and 4) the Wang–Bovik index. The first metric has been widely used in hydrometeorology, and the other three have been proposed in other disciplines for similarity analysis. It is demonstrated that these similarity metrics, in their original formulations, either do not actually have the purported advantage over the traditional Pearson correlation coefficient or are not suitable for some hydrometeorological applications. They are reformulated in this study to address these deficiencies. The resulting modified metrics are unitless, bounded, and proportional to the Pearson correlation coefficient, and three of them have the confirmed advantage of explicitly penalizing for differences in the mean and/or in the variance. Two application examples are used to demonstrate the applicability of these similarity metrics in hydrometeorology. A metavalidation model and a graphical tool (Taylor diagram) are used to evaluate the performances of these similarity metrics. In a case study of analog analysis, the Wang–Bovik index stands out as the best metric for simulation of the human perception of similarity between two-dimensional patterns, whereas the modified Petke index and the traditional root-mean-square distance may perform slightly better than the others in the regions with a very large difference between the variances.

Recognizing and visualizing departures from independence in bivariate data using local Gaussian correlation

It is well known that the traditional Pearson correlation in many cases fails to capture non-linear dependence structures in bivariate data. Other scalar measures capable of capturing non-linear dependence exist. A common disadvantage of such measures, however, is that they cannot distinguish between negative and positive dependence, and typically the alternative hypothesis of the accompanying test of independence is simply "dependence". This paper discusses how a newly developed local dependence measure, the local Gaussian correlation, can be used to construct local and global tests of independence. A global measure of dependence is constructed by aggregating local Gaussian correlation on subsets of $\mathbb{R}^{2}$ , and an accompanying test of independence is proposed. Choice of bandwidth is based on likelihood cross-validation. Properties of this measure and asymptotics of the corresponding estimate are discussed. A bootstrap version of the test is implemented and tried out on both real and simulated data. The performance of the proposed test is compared to the Brownian distance covariance test. Finally, when the hypothesis of independence is rejected, local independence tests are used to investigate the cause of the rejection.

Identifying scale specific controls of soil water storage in a hummocky landscape using wavelet coherency

High spatio-temporal variability of soil water is contributed from different ecohydrological and soil processes operating in different intensities at different scales. Traditional Pearson correlation analysis only examines linear correlation at the measurement scale. In this study, the correlation between soil water storage and its controlling factors was examined at different scales and locations in a hummocky landscape using wavelet coherency. Time domain reflectometry and neutron probe were used to measure soil water storage up to 1.4 m depth along a transect of 576 m long established in a hummocky landscape at St. Denis National Wildlife Area, Saskatchewan, Canada. In spite of visual similarity of the spatial pattern of soil water storage and elevation, the value of Pearson correlation coefficient was very small. However, wavelet coherency identified strong scale- and location-specific correlations between soil water storage and elevation. The total area of significant correlations as calculated from the total number of significant coherencies at different scales and locations was higher between soil water storage and elevation than between soil water storage and any other factors, which indicated a dominant control from elevation on soil water storage in the hummocky landscape. The largest area of significant correlation was observed at large scales (> 70 m), which can be attributed to the alternating knolls and depressions. The relationship between soil water storage and elevation at different scales was persistent at different times of the year or at different seasons with a slight reduction in the magnitude of correlation. The persistent relationship indicated the dominant control from elevation with slight change in the degree of the control. The scale-location specific correlation provides a complete picture on the controls of soil water storage, which was not possible with traditional correlation analysis.

Mixed-model coexpression: calculating gene coexpression while accounting for expression heterogeneity

Motivation: The analysis of gene coexpression is at the core of many types of genetic analysis. The coexpression between two genes can be calculated by using a traditional Pearson's correlation coefficient. However, unobserved confounding effects may cause inflation of the Pearson's correlation so that uncorrelated genes appear correlated. Many general methods have been suggested, which aim to remove the effects of confounding from gene expression data. However, the residual confounding which is not accounted for by these generic correction procedures has the potential to induce correlation between genes. Therefore, a method that specifically aims to calculate gene coexpression between gene expression arrays, while accounting for confounding effects, is desirable.Results: In this article, we present a statistical model for calculating gene coexpression called mixed model coexpression (MMC), which models coexpression within a mixed model framework. Confounding effects are expected to be encoded in the matrix representing the correlation between arrays, the inter-sample correlation matrix. By conditioning on the information in the inter-sample correlation matrix, MMC is able to produce gene coexpressions that are not influenced by global confounding effects and thus significantly reduce the number of spurious coexpressions observed. We applied MMC to both human and yeast datasets and show it is better able to effectively prioritize strong coexpressions when compared to a traditional Pearson's correlation and a Pearson's correlation applied to data corrected with surrogate variable analysis (SVA).Availability: The method is implemented in the R programming language and may be found at http://genetics.cs.ucla.edu/mmc.Contact: nfurlott@cs.ucla.edu; eeskin@cs.ucla.edu

MG-join: detecting phenomena and their correlation in high dimensional data streams

A phenomenon appears in a sensor network when a group of sensors continuously produces similar readings (i.e., data streams) over a period of time. This involves the processing of hundreds and maybe thousands of data streams in real-time. This paper focuses on detecting environmental phenomena and determining possible correlation between such phenomena. This paper proposes an efficient scheme for a detecting and tracking phenomena, e.g., air pollution and oil spills. To achieve fast online response, the proposed algorithms use a Discrete Fourier Transformation (DFT) to reduce the dimensionality of the streams. Each stream is represented by a point in a multidimensional grid in the frequency domain. The algorithm uses an improved unsupervised grid-based clustering technique to detect similar streams and to form clusters. The paper also proposes an efficient algorithm for detecting correlation among phenomena. The proposed algorithm calculates the correlation coefficient in the frequency domain. It makes use of the DFT coefficients that are calculated for detecting the phenomena. The proposed correlation detection algorithm uses only few DFT coefficients in the frequency domain. Experiments on synthetic data streams show that the proposed algorithm for detecting and tracking phenomena is much faster than the DBSCAN clustering technique, which is based on the R-tree index. At the same time, the proposed phenomena detection algorithm produces the same quality as that of the DBSCAN by only using two DFT coefficients in most of the cases. The experimental results also showed that the proposed technique for detecting the correlation among phenomena performs as good as the traditional Pearson correlation formula but it is much faster.

Validity of the GHQ-28 in New Zealand women.

The GHQ-28 was validated against the short PSE in a New Zealand community study of female psychiatric morbidity. The GHQ-28 total scores were significantly correlated with the PSE scores. Higher coefficients were obtained using the scoring method of Goodchild and Duncan-Jones than with the standard scoring method. In this data set, the 3/4 cutoff had the best sensitivity and specificity. The correlations of the GHQ-28 subscales with ICD diagnostic classes and ad hoc PSE sub-scores were also statistically significant. Because the distribution of the GHQ-28 scores is positively skewed, non-parametric statistics may be preferable to the traditional Pearson's correlation coefficient. Overall, the results from this study confirm the GHQ-28 to be a valid and practical screen for presence or absence of psychiatric disorder in New Zealand women.