Inversion Of Covariance Matrices Research Articles

Maximum-likelihood estimators in physics-informed neural networks for high-dimensional inverse problems

Physics-informed neural networks (PINNs) have proven a suitable mathematical scaffold for solving inverse ordinary (ODE) and partial differential equations (PDE). Typical inverse PINNs are formulated as soft-constrained multi-objective optimization problems with several hyperparameters. In this work, we demonstrate that inverse PINNs can be framed in terms of maximum-likelihood estimators (MLE) to allow explicit error propagation from interpolation to the physical model space through a Taylor expansion, removing a hyperparameter from the formulation. We demonstrate the technique through its application to high-dimensional coupled ODEs constrained by differential algebraic equations that are common in transient chemical and biological kinetics. Furthermore, we show that singular-value decomposition (SVD) of the ODE coupling matrix (reaction stoichiometry matrix) provides reduced uncorrelated subspaces in which PINNs solutions can be represented and over which residuals can be projected. Finally, SVD bases serve as preconditioners for the inversion of covariance matrices in this hyperparameter-free robust application of MLE to “kinetics-informed neural networks”.

Semi-Supervised Non-Parametric Bayesian Modelling of Spatial Proteomics.

Understanding sub-cellular protein localisation is an essential component in the analysis of context specific protein function. Recent advances in quantitative mass-spectrometry (MS) have led to high resolution mapping of thousands of proteins to sub-cellular locations within the cell. Novel modelling considerations to capture the complex nature of these data are thus necessary. We approach analysis of spatial proteomics data in a non-parametric Bayesian framework, using K-component mixtures of Gaussian process regression models. The Gaussian process regression model accounts for correlation structure within a sub-cellular niche, with each mixture component capturing the distinct correlation structure observed within each niche. The availability of marker proteins (i.e. proteins with a priori known labelled locations) motivates a semi-supervised learning approach to inform the Gaussian process hyperparameters. We moreover provide an efficient Hamiltonian-within-Gibbs sampler for our model. Furthermore, we reduce the computational burden associated with inversion of covariance matrices by exploiting the structure in the covariance matrix. A tensor decomposition of our covariance matrices allows extended Trench and Durbin algorithms to be applied to reduce the computational complexity of inversion and hence accelerate computation. We provide detailed case-studies on Drosophila embryos and mouse pluripotent embryonic stem cells to illustrate the benefit of semi-supervised functional Bayesian modelling of the data.

A resample-replace lasso procedure for combining high-dimensional markers with limit of detection

In disease screening, a biomarker combination developed by combining multiple markers tends to have a higher sensitivity than an individual marker. Parametric methods for marker combination rely on the inverse of covariance matrices, which is often a non-trivial problem for high-dimensional data generated by modern high-throughput technologies. Additionally, another common problem in disease diagnosis is the existence of limit of detection (LOD) for an instrument – that is, when a biomarker's value falls below the limit, it cannot be observed and is assigned an NA value. To handle these two challenges in combining high-dimensional biomarkers with the presence of LOD, we propose a resample-replace lasso procedure. We first impute the values below LOD and then use the graphical lasso method to estimate the means and precision matrices for the high-dimensional biomarkers. The simulation results show that our method outperforms alternative methods such as either substitute NA values with LOD values or remove observations that have NA values. A real case analysis on a protein profiling study of glioblastoma patients on their survival status indicates that the biomarker combination obtained through the proposed method is more accurate in distinguishing between two groups.

Factor Graph-Based Smoothing Without Matrix Inversion for Highly Precise Localization

We consider the problem of localizing a manned, semi-autonomous, or autonomous vehicle in the environment using information coming from the vehicle's sensors, a problem known as navigation or simultaneous localization and mapping (SLAM) depending on the context. To infer knowledge from sensors' measurements, while drawing on a priori knowledge about the vehicle's dynamics, modern approaches solve an optimization problem to compute the most likely trajectory given all past observations, an approach known as smoothing. Improving smoothing solvers is an active field of research in the SLAM community. Most work is focused on reducing computation load by inverting the involved linear system while preserving its sparsity. The present paper raises an issue which, to the knowledge of the authors, has not been addressed yet: standard smoothing solvers require explicitly using the inverse of sensor noise covariance matrices. This means the parameters that reflect the noise magnitude must be sufficiently large for the smoother to properly function. When matrices are close to singular, which is the case when using high precision modern inertial measurement units (IMU), numerical issues necessarily arise, especially with 32-bits implementation demanded by most industrial aerospace applications. We discuss these issues and propose a solution that builds upon the Kalman filter to improve smoothing algorithms. We then leverage the results to devise a localization algorithm based on fusion of IMU and vision sensors. Successful real experiments using an actual car equipped with a tactical grade high performance IMU and a LiDAR illustrate the relevance of the approach to the field of autonomous vehicles.

A Four Dimensional Variational Data Assimilation Framework for Wind Energy Potential Estimation

In this paper, we propose a Four-Dimensional Variational (4D-Var) data assimilation framework for wind energy potential estimation. The framework is defined as follows: we choose a numerical model which can provide forecasts of wind speeds then, an ensemble of model realizations is employed to build control spaces at observation steps via a modified Cholesky decomposition. These control spaces are utilized to estimate initial analysis increments and to avoid the intrinsic use of adjoint models in the 4D-Var context. The initial analysis increments are mapped back onto the model domain from which we obtain an estimate of the initial analysis ensemble. This ensemble is propagated in time to approximate the optimal analysis trajectory. Wind components are post-processed to get wind speeds and to estimate wind energy capacities. A matrix-free analysis step is derived from avoiding the direct inversion of covariance matrices during assimilation cycles. Numerical simulations are employed to illustrate how our proposed framework can be employed in operational scenarios. A catalogue of twelve Wind Turbine Generators (WTGs) is utilized during the experiments. The results reveal that our proposed framework can properly estimate wind energy potential capacities for all wind turbines within reasonable accuracies (in terms of Root-Mean-Square-Error) and even more, these estimations are better than those of traditional 4D-Var ensemble-based methods. Moreover, large variability (variance of standard deviations) of errors are evidenced in forecasts of wind turbines with the largest rate-capacity while homogeneous variability can be seen in wind turbines with the lowest rate-capacity.

Effective probability distribution approximation for the reconstruction of missing data

Spatially distributed processes can be modeled as random fields. The complex spatial dependence is then incorporated in the joint probability density function. Knowledge of the joint probability density allows predicting missing data. While environmental data often exhibit significant deviations from Gaussian behavior (rainfall, wind speed, and earthquakes being characteristic examples), only a few non-Gaussian joint probability density functions admit explicit expressions. In addition, random field models are computationally costly for big datasets. We propose an “effective distribution” approach which is based on the product of univariate conditional probability density functions modified by local interactions. The effective densities involve local parameters that are estimated by means of kernel regression. The prediction of missing data is based on the median value from an ensemble of simulated states generated from the effective distribution model. The latter can capture non-Gaussian dependence and is applicable to large spatial datasets, since it does not require the storage and inversion of large covariance matrices. We compare the predictive performance of the effective distribution approach with classical geostatistical methods using Gaussian and non-Gaussian synthetic data. We also apply the effective distribution approach to the reconstruction of gaps in large raster data.

Can Virtual Samples Solve Small Sample Size Problem of KISSME in Pedestrian Re-Identification of Smart Transportation?

This work investigates whether virtual samples can solve the small sample size (S3) problem of Keep-It-Simple-and-Straightforward Metric Learning (KISSME) in Pedestrian re-identification (Re-ID). Re-ID is a very challenging and important problem in the field of multi-camera surveillance in smart transportation. Among many hand-crafted ways (not deeply-learned ones) to solve it, KISSME has received great attention. Although it has achieved convincing performance in some applications, it encounters an S3 problem in calculating various classes of covariance matrices whose eigenvalues become too small. Such small eigenvalues cause an instability issue when computing the inverse of covariance matrices, thus resulting in poor Re-ID performance. If we can increase the number of samples, then an S3 problem is alleviated or eliminated. This work makes a hypothesis that virtual samples can do so, and proposes a new algorithm to generate them. It adopts a Genetic Algorithm to generate virtual features (corresponding to virtual samples) based on the dimension-reduced sample features, which eliminates the process of re-extracting features of newly generated virtual samples and save time. It can clearly increase the magnitude of otherwise small eigenvalues, helps one perform the accurate estimation of the inverse of various covariance matrices and finally alleviates the S3 problem. Experimental results based on a commonly-used database confirm that the proposed method can significantly improve the matching rate of pedestrian Re-ID, which fully shows that virtual samples are indeed effective for alleviating the S3 problem in pedestrian Re-ID.

Fast hybrid Bayesian integrative learning of multiple gene regulatory networks for type 1 diabetes.

Motivated by the study of the molecular mechanism underlying type 1 diabetes with gene expression data collected from both patients and healthy controls at multiple time points, we propose a hybrid Bayesian method for jointly estimating multiple dependent Gaussian graphical models with data observed under distinct conditions, which avoids inversion of high-dimensional covariance matrices and thus can be executed very fast. We prove the consistency of the proposed method under mild conditions. The numerical results indicate the superiority of the proposed method over existing ones in both estimation accuracy and computational efficiency. Extension of the proposed method to joint estimation of multiple mixed graphical models is straightforward.

A hybrid of Bayesian approach based global search with clustering aided local refinement

We propose a global optimization algorithm in which the P-algorithm is hybridized with a clustering aided local search. The proposed algorithm is free from the computational burden typical for the known implementations of Bayesian algorithms, i.e. the inversion of high dimensional covariance matrices, and the solution of inner non-convex optimization problems. Clustering aided switching between the global and local search rationally combines advantages of both strategies: the basins of potential global minima indicated by the clustering are excluded from the global search, and the potential minima are refined by a local search method. We complete the paper with numerical experiments that illustrate the performance achieved.

Local inversion-free estimation of spatial Gaussian processes

Maximizing the likelihood has been widely used for estimating the unknown covariance parameters of spatial Gaussian processes. However, evaluating and optimizing the likelihood function can be computationally intractable, particularly for large number of (possibly) irregularly spaced observations, due to the need to handle the inverse of ill-conditioned and large covariance matrices. Extending the “inversion-free” method of Anitescu, Chen and Stein [1], we investigate a broad class of covariance parameter estimation based on inversion-free surrogate losses and block diagonal approximation schemes of the covariance structure. This class of estimators yields a spectrum for negotiating the trade-off between statistical accuracy and computational cost. We present fixed-domain asymptotic properties of our proposed method, establishing $\sqrt{n}$-consistency and asymptotic normality results for isotropic Matern Gaussian processes observed on a multi-dimensional and irregular lattice. Simulation studies are also presented for assessing the scalability and statistical efficiency of the proposed algorithm for large data sets.

A Powerful Bayesian Test for Equality of Means in High Dimensions

ABSTRACTWe develop a Bayes factor-based testing procedure for comparing two population means in high-dimensional settings. In ‘large-p-small-n” settings, Bayes factors based on proper priors require eliciting a large and complex p × p covariance matrix, whereas Bayes factors based on Jeffrey’s prior suffer the same impediment as the classical Hotelling T2 test statistic as they involve inversion of ill-formed sample covariance matrices. To circumvent this limitation, we propose that the Bayes factor be based on lower dimensional random projections of the high-dimensional data vectors. We choose the prior under the alternative to maximize the power of the test for a fixed threshold level, yielding a restricted most powerful Bayesian test (RMPBT). The final test statistic is based on the ensemble of Bayes factors corresponding to multiple replications of randomly projected data. We show that the test is unbiased and, under mild conditions, is also locally consistent. We demonstrate the efficacy of the approach through simulated and real data examples. Supplementary materials for this article are available online.

Kernelized random KISS metric learning for person re-identification

Abstract Person re-identification is critical for human tracking in the video surveillance which has attracted more and more attention in recent years. Various recent approaches have made great progress in re-identification performance using metric learning techniques and among them, Keep It Simple and Straightforward (KISS) metric learning method has shown remarkably importance because of its simpleness and high-efficiency. The KISS method is based on an assumption that the differences between feature pairs obey the Gaussian distribution. However, for most existing features of person re-identification, the distributions of differences between feature pairs are irregular and undulant. Therefore, prior to the Guassian based metric learning step, it's important to augment the Guassian distribution of data without losing discernment. Moreover, most metric learning methods were greatly influenced by the small sample size (SSS) problem and the KISS method is no exception, which causing the inexistence of inverse of covariance matrices. To solve the above two problems, we present Kernelized Random KISS (KRKISS) metric learning method. By transforming the original features into kernelized features, the differences between feature pairs can better fit the Gaussian distribution and thus they can be more suitable for the Guassian assumption based models. To solve the inverse of covariance matrix estimation problem, we apply a random subspace ensemble method to obtain exact estimation of covariance matrix by randomly selecting and combining several different subspaces. In each subspace, the influence of SSS problem can be minimized. Experimental results on three challenging person re-identification datasets demonstrate that the KRKISS method significantly improves the KISS method and achieves better performance than most existing metric learning approaches.

Numerical instability of calculating inverse of spatial covariance matrices

Computing an inverse of a covariance matrix is a common computational component in Statistics. For example, Gaussian likelihood function includes the inverse of a covariance matrix. For the computation of the inverse of a spatial covariance matrix, numerically unstable results can arise when the observation locations are getting denser. In this paper, we investigate when computational instability in calculating the inverse of a spatial covariance matrix makes maximum likelihood estimator unreasonable for a Matérn covariance model. Also, some possible approaches to relax such computational instability are discussed.

Improved efficiency of maximum likelihood analysis of time series with temporally correlated errors

Most time series of geophysical phenomena have temporally correlated errors. From these measurements, various parameters are estimated. For instance, from geodetic measurements of positions, the rates and changes in rates are often estimated and are used to model tectonic processes. Along with the estimates of the size of the parameters, the error in these parameters needs to be assessed. If temporal correlations are not taken into account, or each observation is assumed to be independent, it is likely that any estimate of the error of these parameters will be too low and the estimated value of the parameter will be biased. Inclusion of better estimates of uncertainties is limited by several factors, including selection of the correct model for the background noise and the computational requirements to estimate the parameters of the selected noise model for cases where there are numerous observations. Here, I address the second problem of computational efficiency using maximum likelihood estimates (MLE). Most geophysical time series have background noise processes that can be represented as a combination of white and power-law noise, 1/f^{alpha } with frequency, f. With missing data, standard spectral techniques involving FFTs are not appropriate. Instead, time domain techniques involving construction and inversion of large data covariance matrices are employed. Bos et al. (J Geod, 2013. doi:10.1007/s00190-012-0605-0) demonstrate one technique that substantially increases the efficiency of the MLE methods, yet is only an approximate solution for power-law indices >1.0 since they require the data covariance matrix to be Toeplitz. That restriction can be removed by simply forming a data filter that adds noise processes rather than combining them in quadrature. Consequently, the inversion of the data covariance matrix is simplified yet provides robust results for a wider range of power-law indices.

Asymptotic normal tests for integration in panels with cross-dependent units

The asymptotically normal, regression-based LM integration test is adapted for panels with correlated units. The N different units may be integrated of different (fractional) orders under the null hypothesis. The paper first reviews conditions under which the test statistic is asymptotically (as T→∞) normal in a single unit. Then we adopt the framework of seemingly unrelated regression [SUR] for cross-correlated panels, and discuss a panel test statistic based on the feasible generalized least squares [GLS] estimator, which follows a χ2(N) distribution. Third, a more powerful statistic is obtained by working under the assumption of equal deviations from the respective null in all units. Fourth, feasible GLS requires inversion of sample covariance matrices typically imposing T>N; in addition we discuss alternative covariance matrix estimators for T<N. The usefulness of our results is assessed in Monte Carlo experimentation.

Application of FFT-based Algorithms for Large-Scale Universal Kriging Problems

Looking at kriging problems with huge numbers of estimation points and measurements, computational power and storage capacities often pose heavy limitations to the maximum manageable problem size. In the past, a list of FFT-based algorithms for matrix operations have been developed. They allow extremely fast convolution, superposition and inversion of covariance matrices under certain conditions. If adequately used in kriging problems, these algorithms lead to drastic speedup and reductions in storage requirements without changing the kriging estimator. However, they require second-order stationary covariance functions, estimation on regular grids, and the measurements must also form a regular grid. In this study, we show how to alleviate these rather heavy and many times unrealistic restrictions. Stationarity can be generalized to intrinsicity and beyond, if decomposing kriging problems into the sum of a stationary problem and a formally decoupled regression task. We use universal kriging, because it covers arbitrary forms of unknown drift and all cases of generalized covariance functions. Even more general, we use an extension to uncertain rather than unknown drift coefficients. The sampling locations may now be irregular, but must form a subset of the estimation grid. Finally, we present asymptotically exact but fast approximations to the estimation variance and point out application to conditional simulation, cokriging and sequential kriging. The drastic gain in computational and storage efficiency is demonstrated in test cases. Especially high-resolution and data-rich fields such as rainfall interpolation from radar measurements or seismic or other geophysical inversion can benefit from these improvements.

Augmenting the bootstrap to analyze high dimensional genomic data

The data produced by high-throughput genomic techniques are often high dimensional and undersampled. In these settings, statistical analyses that require the inversion of covariance matrices, such as those pursuing supervised dimension reduction or the assessment of interdependence structures, are problematic. In this article we show how the idea of adding noise to the bootstrap, pioneered by Efron, and Silverman and Young, in the late seventies and eighties, can be used to overcome undersampling and effectively estimate the inverse covariance matrix for data sets in which the number of observations is small relative to the number of variables. We demonstrate the performance of this approach, which we call augmented bootstrap, on simulated data and on data derived from genomic DNA sequences and microarray experiments.

A Covariance Matrix Inversion Problem arising from the Construction of Phylogenetic Trees

AbstractWe describe an efficient algorithm for the inversion of covariance matrices that arise in the context of phylogenetic tree construction. Phylogenetic trees describe the evolutionary relationships between species, and their construction is computationally demanding. Many approaches involve the symmetric matrix of evolutionary distances between species. Regarding these distances as random variables, the corresponding set of variances and covariances form a rank-4 tensor, and the inner-product defined by its inverse can be used to assign statistical scores to candidate trees. We describe a natural set of assumptions for the phylogenetic tree under construction, and show how under these assumptions the covariance tensor for a tree with n leaves can be inverted in O(n2) operations. In addition to presenting the inversion algorithm, we hope this article will open algebraic and computational problems from the field of phylogeny to a wider audience.

Correlation filters for target detection in a Markov model background clutter

The performance of distortion-invariant correlation filters in the presence of background clutter is addressed. Background images are modeled as Markov noise processes, and a synthesis procedure for the optimal filter is described. It is shown that spatially filtering the training set images eliminates the need for the inversion of large noise covariance matrices, thus leading to a computationally efficient filter realization. The effect of errors (in the estimation of clutter correlation coefficient) on filter performance is theoretically analyzed, and a bound on the relative degradation of the SNR due to such errors is presented.

Estimation - Correlation, Modeling and Identification in Adaptive Array Processors

This paper studies the system modeling and identification issues that arise in implementation of maximum likelihood detectors and adaptive array processors of signals that have propagated over randomly time-varying propagation and scattering channels. A multidimensional estimator correlator processor is used to combine the ideas of system modeling and identification with detection theory and adaptive array processing. Part of the estimator structure can be obtained from received data by the usual adaptive array processing techniques (estimation and inversion of noise covariance matrices), but the channel output estimator requires prior knowledge of the signal part of the channel output covariance matrix. This requires modeling and identification of deterministic and stochastic propagation and scattering operators. A number of convenient matrix representations for these operators have been developed. Identification of stochastic operators is based on the vector Karhunen-Loeve expansion. It is shown that in the special case of wide-sense stationary stochastic processes and periodic signal vector, orthonormal basis and eigenvalues for channel modeling and identification can be determined by discrete Fourier transform and singular value decomposition.