Estimated Correlation Matrix Research Articles

Bayesian inference for link travel time correlation of a bus route

Estimation of link travel time correlation of a bus route is essential to many bus operation applications. Link travel time on a bus route could exhibit complex correlation structures, such as long-range correlations, negative correlations, and time-varying correlations. This paper develops a Bayesian Gaussian model to estimate the link travel time correlation matrix of a bus route using smart-card-like data. Our method overcomes the small-sample-size problem in correlation matrix estimation by borrowing/integrating those incomplete observations from other bus routes. We first conduct a synthetic experiment and results show that the proposed method produces an accurate estimation for correlations with credible intervals. Next, we perform experiments on a real-world bus route with in-out-stop record data; results show that both local and long-range correlations exist on this bus route. Finally, we demonstrate an application of using the estimated covariance matrix to make probabilistic forecasting of link and trip travel time.

Rank-based correlation matrix estimation for high dimensional microbiome data

In recent years, the progress made in high-throughout sequencing techniques supports the analysis of microbial communities greatly. The fact that real-world microbiome data lie in a high-dimensional simplex makes conventional statistical analysis unreliable owing to the sum-to-one constraint. Motivated by the urgent requirements of directly inferring the dependence pattern and dependence intensity simultaneously among microbial taxa, we address the challenge of correlation matrix estimation for this kind of data. Assuming that the unobserved log-basis random vectors follow the elliptical distribution, a robust method built on the Kendall's tau statistic is developed to estimate the basis generalized correlation matrix. Regardless of the existence of the covariance, the basis generalized correlation matrix is always a reliable measure of dependence structure. Theoretically, we establish the convergence rate of the proposed estimator under the spectral norm. Support recovery is discussed as well. Empirically, simulation studies demonstrate the outstanding numerical performance of our method. We also apply our method to analyse a microbiome dataset from human gut. We provide all the codes at https://github.com/lwfwhunanhero/CRCE.

Direct estimation and inference of higher-level correlations from lower-level measurements with applications in gene-pathway and proteomics studies.

This paper tackles the challenge of estimating correlations between higher-level biological variables (e.g. proteins and gene pathways) when only lower-level measurements are directly observed (e.g. peptides and individual genes). Existing methods typically aggregate lower-level data into higher-level variables and then estimate correlations based on the aggregated data. However, different data aggregation methods can yield varying correlation estimates as they target different higher-level quantities. Our solution is a latent factor model that directly estimates these higher-level correlations from lower-level data without the need for data aggregation. We further introduce a shrinkage estimator to ensure the positive definiteness and improve the accuracy of the estimated correlation matrix. Furthermore, we establish the asymptotic normality of our estimator, enabling efficient computation of P-values for the identification of significant correlations. The effectiveness of our approach is demonstrated through comprehensive simulations and the analysis of proteomics and gene expression datasets. We develop the R package highcor for implementing our method.

Multivariate probit linear mixed models for multivariate longitudinal binary data.

When analyzing multivariate longitudinal binary data, we estimate the effects on the responses of the covariates while accounting for three types of complex correlations present in the data. These include the correlations within separate responses over time, cross-correlations between different responses at different times, and correlations between different responses at each time point. The number of parameters thus increases quadratically with the dimension of the correlation matrix, making parameter estimation difficult; the estimated correlation matrix must also meet the positive definiteness constraint. The correlation matrix may additionally be heteroscedastic; however, the matrix structure is commonly considered to be homoscedastic and constrained, such as exchangeable or autoregressive with order one. These assumptions are overly strong, resulting in skewed estimates of the covariate effects on the responses. Hence, we propose probit linear mixed models for multivariate longitudinal binary data, where the correlation matrix is estimated using hypersphere decomposition instead of the strong assumptions noted above. Simulations and real examples are used to demonstrate the proposed methods. An open source R package, BayesMGLM, is made available on GitHub at https://github.com/kuojunglee/BayesMGLM/ with full documentation to produce the results.

Performance evaluation of sound source localisation and tracking methods using multiple drones

This study evaluates methods to improve sound source localisation and tracking performance using microphone arrays from multiple drones. Previous studies have demonstrated the feasibility of multi-drone sound source tracking by triangulating their respective direction estimations using the MUltiple SIgnal Classification (MUSIC) algorithm. In addition, we evaluate techniques to reduce the influence of rotor noise. Namely, this includes a Wiener filter that utilises rotor noises' power spectral density (PSD) estimations and a noise correlation matrix (NCM) estimation scheme exploiting the geometries of the drone and the microphone array. This is to reduce the rotor noise's influence on the microphone array input correlation matrix prior to localising with MUSIC. Results show that applying the rotor noise reduction method improves localisation and tracking accuracy under a wider range of signal-to-noise ratios, regardless of the sound source tracking method.

Simultaneous estimation of the intermediate correlation matrix for arbitrary marginal densities.

A popular approach to the simulation of multivariate, non-normal data in the social sciences is to define a multivariate normal distribution first, and then alter its lower-dimensional marginals to achieve the shape of the distribution intended by the researchers. A consequence of this process is that the correlation structure is altered, so further methods are needed to specify an intermediate correlation matrix in the multivariate normal distribution step. Most of the techniques available in the literature estimate this intermediate correlation matrix bivariately (i.e., correlation by correlation), risking the possibility of generating a non-positive definite matrix. The present article addresses this issue by offering an algorithm that estimates all elements of the intermediate correlation matrix simultaneously, through stochastic approximation. A small simulation study demonstrates the feasibility of the present method to induce the correlation structure both in simulated and empirical data.

A novel approach to denoising correlation matrices with applications to global portfolio management with a large number of assets

We introduce a new approach to denoising correlation matrices that imposes a block structure with a fixed block-dependent pair-wise correlation within each block and a constant correlation specified for each pair of blocks. We characterize the eigenvalue spectrum and modify the Marchenko-Pastur distribution of eigenvalues. We present approximate analytic solutions for the inverse problem of determining the block sizes and the correlation parameters under a broad set of assumptions. Our solution is based on a novel unsupervised approach for improving correlation matrix estimation by a functional transformation of the original data rather than popular shrinkage estimation or standard random-matrix-theory-based techniques. However, a correlation matrix produced by our method can serve as an attractive target correlation matrix within shrinkage frameworks. Our correlation matrix denoising method has broad applications for global portfolio management with a large number of assets from many diverse asset classes.

Separation-based parameterization strategies for estimation of restricted covariance matrices in multivariate model systems

Many multivariate model systems involve the estimation of a covariance matrix that must be positive-definite. A common strategy to ensure positive definiteness of the covariance matrix is through the use of a Cholesky parameterization of the covariance matrix. However, several model systems require imposing restrictions on the elements of the covariance elements. For instance, modelling systems may require fixing some (or all) of the diagonal elements in the covariance matrix to unity due to identification considerations. However, imposing such restrictions using the traditional Cholesky decomposition approach is not feasible and requires the additional parameterization of the Cholesky elements.In this paper, we explore a separation-based strategy with spherical parameterization of the Cholesky matrix to impose restrictions on the covariance matrix. Importantly, using this separation-based parameterization strategy, we also explore the possibility of restricting some covariance (or correlation) terms to zero. The effectiveness of the proposed strategy is assessed through extensive simulation experiments. The results from the simulation experiments highlight better performance of the separation-based strategy in terms of recovery of model parameters – particularly those in the covariance matrix, than the traditional Cholesky parameterization approach. Finally, the proposed strategy is implemented in a joint multivariate binary probit ordered probit model system to analyze the usage (and the extent of use) of non-private modes of transportation in Bengaluru, India. In doing so, the proposed strategy is implemented to restrict several correlations to zero, thus avoiding the estimation of a profligate correlation matrix and substantially easing the estimation process.

КРИТЕРІАЛЬНІ ПОКАЗНИКИ ЯКОСТІ ОБРОБКИ МУЛЬТИСЕНСОРНОЇ ІНФОРМАЦІЇ В УМОВАХ ДЕСТАБІЛІЗУЮЧИХ ВПЛИВІВ

The approach to determining the criterion indicators of the quality of digital information processing in the multi-sensor information-controlled system of the car under the influence of destabilizing factors is considered. It was established that the process of processing information that enters the information-controlled system in digital form from various sensors involves the calculation of discrete convolutions of matrices of relevant operators and arrays of initial data. This process includes procedures for forming an array of responses to the observed information flow, an array of responses to the restored (restored) information flow, and forming conditions for restoration (restoration) of the observed information flow. The approach to assessing the quality of digital information processing in terms of determining subjective and objective assessments and methods of obtaining them is analyzed. The basis of these methods is the procedure of direct inversion of the estimated correlation matrix of random realizations of the observed multidimensional information process. Systems with an inverse operator for processing a multidimensional information array are quite sensitive to destabilizing influences: to random perturbations of the coefficients of the recovery (processing) operator and internal noise of the system along with internal system perturbations. The need to take into account destabilizing influences in the process of processing digital information in the car's information-controlled system necessitates the development of a general methodology for studying the influence of destabilizing factors on the quality of image processing. Based on the application of inverse synthesis methods, analytical expressions for criterion quality indicators of digital image processing are proposed, which are not limited to the upper limit of quality assessment and allow analytical evaluation and quantitative determination of the quality level of discrete images restored by direct inversion at the stage of system synthesis. Keywords: car, information-controlled system, image, restoration, restoration operator, criterion indicators, multisensory information, quality of information processing. internal noise, destabilizing factors.

Estimation of high-dimensional sparse cross correlation matrix

Estimation of high-dimensional sparse cross correlation matrix

State space partitioning based on constrained spectral clustering for block particle filtering

To overcome the curse of dimensionality of particle filter (PF), the block PF (BPF) partitions the state space into several blocks of smaller dimension so that the correction and resampling steps can be performed independently on each block. Despite its potential performance, this approach has a practical limitation. BPF requires an additional input compared to classical PF: the partition from which blocks are defined. A poor choice will not offer the expected performance gain.In this paper, we formulate the partitioning problem as a clustering problem and propose a data-driven partitioning method based on constrained spectral clustering (CSC) to automatically provide an appropriate partition.We design a generalized BPF that contains two new steps: (i) estimation of the state vector correlation matrix from particles, (ii) CSC using this estimate to determine blocks. The proposed method succeeds in providing an online partition into blocks restricted in size and grouping the most correlated state variables. This partition allows the BPF to escape the curse of dimensionality by reducing the variance of the filtering distribution estimate while limiting the level of bias. Since our approach relies on particles that are already necessary to generate as part of BPF, the computation overhead is limited.

High-dimensional correlation matrix estimation for general continuous data with Bagging technique

High-dimensional covariance matrix estimation plays a central role in multivariate statistical analysis. It is well-known that the sample covariance matrix is singular when the sample size is smaller than the dimension of the variable, but the covariance estimate must be positive-definite. This motivates some modifications of the sample covariance matrix to preserve its efficient estimation of pairwise covariance. In this paper, we modify the sample correlation matrix using the Bagging technique. The proposed Bagging estimator is flexible for general continuous data. Under some mild conditions, we show theoretically that the Bagging estimator can ensure positive-definiteness with probability one in finite samples. We also prove the consistency of the bootstrap estimator of Pearson correlation and the consistency of our Bagging estimator when the dimension p is fixed. Simulation results and a real application are provided to demonstrate that our method strikes a better balance between RMSE and likelihood, and is more robust, than other existing estimators.

A combinatorial optimization approach to scenario filtering in portfolio selection

Recent studies stressed the fact that covariance matrices computed from empirical financial time series appear to contain a high amount of noise. This makes the classical Markowitz Mean–Variance Optimization model unable to correctly evaluate the performance associated to selected portfolios. Since the Markowitz model is still one of the most used practitioner-oriented tool, several filtering methods have been proposed in the literature to overcome the problem. Among them, the two most promising ones refer to the Random Matrix Theory and to the Power Mapping strategy. The basic idea of these methods is to transform the estimated correlation matrix before applying the Mean–Variance Optimization model. However, experimental analysis shows that these two strategies are not always effective when applied to real financial datasets.In this paper we propose a new filtering method based on Quadratic Programming. We develop a Mixed Integer Quadratic Programming model, which is able to filter those observations that may affect the performance of the selected portfolio. We discuss the properties of this new model and test it on some real financial datasets. We compare the out-of-sample performance of our portfolios with the one of the portfolios provided by the two above mentioned alternative filtering methods giving evidence that our method outperforms them. Although our model can be solved efficiently with standard optimization solvers, the computational burden increases for large datasets. To solve also these problems, we propose a heuristic procedure, which, on the basis of our empirical results, shows to be both efficient and effective.

Low-Complexity Linear Diversity-Combining Detector for MIMO-OTFS

This paper presents a low complexity detector for multiple-input multiple-output (MIMO) systems based on the recently proposed orthogonal time frequency space (OTFS) modulation. In the proposed detector, the copies of the transmitted symbol-vectors received through the different diversity branches (propagation paths and receive antennas) are linearly combined using the maximum ratio combining (MRC) technique to iteratively improve the signal to interference plus noise ratio (SINR) at the output of the combiner. To alleviate the performance degradation due to spatial correlation at the receiver antennas, we present a sample-based method to estimate such correlation and find the optimized combining weights for MRC from the estimated correlation matrix. The detector performance and complexity improve over the linear minimum mean square error (LMMSE) and message passing (MP) detectors proposed in the literature for MIMO-OTFS.

Large dynamic covariance matrices: Enhancements based on intraday data

Multivariate GARCH models do not perform well in large dimensions due to the so-called curse of dimensionality. The recent DCC-NL model of Engle et al. (2019) is able to overcome this curse via nonlinear shrinkage estimation of the unconditional correlation matrix. In this paper, we show how performance can be increased further by using open/high/low/close (OHLC) price data instead of simply using daily returns. A key innovation, for the improved modeling of not only dynamic variances but also of dynamic correlations, is the concept of a regularized return, obtained from a volatility proxy in conjunction with a smoothed sign of the observed return.

State Space Partitioning Based on Constrained Spectral Clustering for Block Particle Filtering

The particle filter (PF) is a powerful inference tool widely used to estimate the filtering distribution in non-linear and/or non-Gaussian problems. To overcome the curse of dimensionality of PF, the block PF (BPF) inserts a blocking step to partition the state space into several subspaces or blocks of smaller dimension so that the correction and resampling steps can be performed independently on each subspace. Using blocks of small size reduces the variance of the filtering distribution estimate, but in turn the correlation between blocks is broken and a bias is introduced. When the dependence relationships between state variables are unknown, it is not obvious to decide how to split the state space into blocks and a significant error overhead may arise from a poor choice of partitioning. In this paper, we formulate the partitioning problem in the BPF as a clustering problem and we propose a state space partitioning method based on spectral clustering (SC). We design a generalized BPF algorithm that contains two new steps: (i) estimation of the state vector correlation matrix from predicted particles, (ii) SC using this estimate as the similarity matrix to determine an appropriate partition. In addition, a constraint is imposed on the maximal cluster size to prevent SC from providing too large blocks. We show that the proposed method can bring together in the same blocks the most correlated state variables while successfully escaping the curse of dimensionality.

Overcoming the impacts of two-step batch effect correction on gene expression estimation and inference.

Nonignorable technical variation is commonly observed across data from multiple experimental runs, platforms, or studies. These so-called batch effects can lead to difficulty in merging data from multiple sources, as they can severely bias the outcome of the analysis. Many groups have developed approaches for removing batch effects from data, usually by accommodating batch variables into the analysis (one-step correction) or by preprocessing the data prior to the formal or final analysis (two-step correction). One-step correction is often desirable due it its simplicity, but its flexibility is limited and it can be difficult to include batch variables uniformly when an analysis has multiple stages. Two-step correction allows for richer models of batch mean and variance. However, prior investigation has indicated that two-step correction can lead to incorrect statistical inference in downstream analysis. Generally speaking, two-step approaches introduce a correlation structure in the corrected data, which, if ignored, may lead to either exaggerated or diminished significance in downstream applications such as differential expression analysis. Here, we provide more intuitive and more formal evaluations of the impacts of two-step batch correction compared to existing literature. We demonstrate that the undesired impacts of two-step correction (exaggerated or diminished significance) depend on both the nature of the study design and the batch effects. We also provide strategies for overcoming these negative impacts in downstream analyses using the estimated correlation matrix of the corrected data. We compare the results of our proposed workflow with the results from other published one-step and two-step methods and show that our methods lead to more consistent false discovery controls and power of detection across a variety of batch effect scenarios. Software for our method is available through GitHub (https://github.com/jtleek/sva-devel) and will be available in future versions of the $\texttt{sva}$ R package in the Bioconductor project (https://bioconductor.org/packages/release/bioc/html/sva.html).

A new correlated polyhedral uncertainty set for robust optimization

Robust optimization approaches are commonly applied in solving a problem with uncertainty. One of the main issues in dealing with uncertainties is the correlations among uncertain parameters. This subject is rarely considered by the researchers and hence the proposed robust approaches normally lead to the solutions which include perturbations with low probability of occurrence. This results in solutions with over conservatism. In this research, an estimation of correlation matrix is applied in order to provide a new uncertainty set that includes probable perturbations. Furthermore in order to trade-off between optimality and level of robustness, a decision making parameter is applied to formulate the corresponding robust counterpart. In order to study the performance of the proposed method, an uncertain optimization problem with correlated uncertain coefficients is solved. Results of the study reveal that the proposed model has superior performance than that of the existing robust approaches.

Information-Based Node Selection for Joint PCA and Compressive Sensing-Based Data Aggregation

Recently it has been shown that when Principal Component Analysis is applied as a dictionary learning technique to Compressive Sensing-based data aggregation, using a Deterministic Node Selection method for data collection in Wireless Sensor Networks can outperform Random Node Selection ones. In this paper, a new scheduling method for selection of measured nodes in a data collection round, called “Information-Based Deterministic Node Selection”, is proposed. Simulation results for synthetic and real data sets show that the proposed method outperforms a reference DNS method in terms of energy consumption per reconstruction error. Correlation (or covariance) matrix estimation is necessary for DNS strategies which are accomplished by gathering data from all network nodes in a few initial time slots of collection rounds. In this regard, we also propose the use of a particular type of shrinkage estimator in preference to the standard correlation matrix estimator. With the aid of the new estimator, we can obtain data correlations with the same accuracy of standard estimator while we need less number of observations. Our numerical experiments demonstrate that when the number of measured nodes is less than 50% of the total nodes, using shrinkage estimator causes extra energy savings in sensor nodes.

High-dimensional correlation matrix estimation for Gaussian data: a Bayesian perspective

High-dimensional correlation matrix estimation for Gaussian data: a Bayesian perspective