Estimation Of Inverse Covariance Matrix Research Articles

Sparse precision matrix estimation under lower polynomial moment assumption

Precision matrix (inverse covariance matrix) estimation is a rising challenge in contemporary applications while dealing with high‐dimensional data. This paper focuses on large‐scale precision matrix of the random vector that only has lower polynomial moments. We mainly investigate upper bounds of the proposed estimator under the spectral norm in terms of the probability and mean estimation respectively. It is shown that the data‐driven estimator is fully adaptive and achieves the same optimal convergence order as under Gaussian assumption on the data. Simulation studies further support our theoretical claims.

Inverse Covariance Matrix Estimation for Low-Complexity Closed-Loop DPD Systems: Methods and Performance

In this article, we study closed-loop digital predistortion (DPD) systems and associated learning algorithms. Specifically, we propose various low-complexity approaches to estimate and manipulate the inverse of the input data covariance matrix (CM) and combine them with the so-called self-orthogonalized (SO) learning rule. The inherent simplicity of the SO algorithm, combined with the proposed solutions, allows for remarkably reduced complexity in the DPD system while maintaining similar linearization performance compared to other state-of-the-art methods. This is demonstrated with thorough over-the-air (OTA) mmW measurement results at 28 GHz, incorporating a state-of-the-art 64-element active antenna array, and very wide channel bandwidths up to 800 MHz. In addition, complexity analyses are carried out, which together with the measured linearization performance demonstrates favorable performance–complexity tradeoffs in linearizing mmW active array transmitters through the proposed solutions. The techniques can find application in systems where the power amplifier (PA) nonlinearities are time-varying and thus frequent or even constant updating of the DPD is required, good examples being mmW adaptive antenna arrays as well as terminal transmitters in 5G and beyond networks.

Sparse inverse covariance matrix estimation via the -norm with Tikhonov regularization**The research is supported by the Special Project on High-performance Computing under the National Key R&D Program (Grant No. 2016YFB0200602), by the Natural Science Foundation of China under Grant Nos. 11701189 and 11771464.

Sparse inverse covariance matrix estimation is a fundamental problem in a Gaussian network model and has attracted much attention in the last decade. A widely used approach for estimating the sparse inverse covariance matrix is the regularized negative log-likelihood minimization, where the regularization term promotes the sparsity of the inverse covariance matrix. Although the -norm is the most popular sparsity promoting function due to its convexity, it may not result in sufficiently satisfied estimations. The -norm is the most natural function to measure sparsity and the -norm regularized negative log-likelihood minimization has been demonstrated to produce favorable results. However, when the sample covariance matrix is singular, the -norm problem has no optimal solution. In this paper, we propose to estimate the sparse inverse covariance matrix via simultaneously using the -norm and the Tikhonov regularization. The resulting minimization problem is guaranteed to have optimal solutions. To overcome the numerical difficulty caused by the -norm, we utilize the penalty decomposition approach. We prove the asymptotic approximation and exact approximation to local (resp., global) minimizers of the original problem by those of the penalty decomposition problem. Then, local minimizers of the penalty decomposition problem are characterized as fixed-points of a system of fixed-point equations with the help of proximity operators. Based on these theoretical results, we design for solving the proposed minimization problem a penalty decomposition fixed-point proximity algorithm and provide the convergence analysis of the algorithm. Numerical results indicate that the proposed algorithm outperforms the compared state-of-the-art algorithms in terms of accuracy.

A permutation-based Bayesian approach for inverse covariance estimation

Covariance estimation and selection for multivariate datasets in a high-dimensional regime is a fundamental problem in modern statistics. Gaussian graphical models are a popular class of models used for this purpose. Current Bayesian methods for inverse covariance matrix estimation under Gaussian graphical models require the underlying graph and hence the ordering of variables to be known. However, in practice, such information on the true underlying model is often unavailable. We therefore propose a novel permutation-based Bayesian approach to tackle the unknown variable ordering issue. In particular, we utilize multiple maximum a posteriori estimates under the DAG-Wishart prior for each permutation, and subsequently construct the final estimate of the inverse covariance matrix. The proposed estimator has smaller variability and yields order-invariant property. We establish posterior convergence rates under mild assumptions and illustrate that our method outperforms existing approaches in estimating the inverse covariance matrices via simulation studies.

An efficient algorithm for sparse inverse covariance matrix estimation based on dual formulation

Estimating large and sparse inverse covariance matrix plays a fundamental role in modern multivariate analysis, because the zero entries capture the conditional independence between pairs of variables given all other variables. This estimation task can be realized by penalizing the maximum likelihood estimation with an adaptive group lasso penalty imposed directly on the elements of the inverse, which allows the estimated to have a blockwise sparse structure that is particularly useful in some applications. In the paper, we are particularly interested in studying the implementation of optimization algorithms for minimizing a class of log-determinant model. This considered minimization model, one the one hand, contains a large number of popular sparse models as special cases, but on the other hand, it poses more challenges especially in high-dimensional situations. Instead of targeting the challenging optimization problem directly, we employ the symmetric Gauss–Seidel (sGS) iteration based alternating direction method of multipliers (ADMM) to tackle the 3-block nonsmooth dual program. By choosing an appropriate proximal term, it was shown that the implemented sGS-ADMM is equivalent to the 2-block ADMM, so its convergence is followed directly from some existing theoretical results. Numerical experiments on synthetic data and real data sets, including the performance comparisons with the directly extended ADMM, demonstrate that the implemented algorithm is effective in estimating large and sparse inverse covariance matrices.

DBSDA : Lowering the Bound of Misclassification Rate for Sparse Linear Discriminant Analysis via Model Debiasing.

Linear discriminant analysis (LDA) is a well-known technique for linear classification, feature extraction, and dimension reduction. To improve the accuracy of LDA under the high dimension low sample size (HDLSS) settings, shrunken estimators, such as Graphical Lasso, can be used to strike a balance between biases and variances. Although the estimator with induced sparsity obtains a faster convergence rate, however, the introduced bias may also degrade the performance. In this paper, we theoretically analyze how the sparsity and the convergence rate of the precision matrix (also known as inverse covariance matrix) estimator would affect the classification accuracy by proposing an analytic model on the upper bound of an LDA misclassification rate. Guided by the model, we propose a novel classifier, DBSDA , which improves classification accuracy through debiasing. Theoretical analysis shows that DBSDA possesses a reduced upper bound of misclassification rate and better asymptotic properties than sparse LDA (SDA). We conduct experiments on both synthetic datasets and real application datasets to confirm the correctness of our theoretical analysis and demonstrate the superiority of DBSDA over LDA, SDA, and other downstream competitors under HDLSS settings.

An Ensemble Kalman Filter Implementation Based on Modified Cholesky Decomposition for Inverse Covariance Matrix Estimation

This paper develops an efficient implementation of the ensemble Kalman filter based on a modified Cholesky decomposition for inverse covariance matrix estimation. This implementation is named EnKF-MC. Background errors corresponding to distant model components with respect to some radius of influence are assumed to be conditionally independent. This allows one to obtain sparse estimators of the inverse background error covariance matrix. The computational effort of the proposed method is discussed and different formulations based on various matrix identities are provided. Furthermore, an asymptotic proof of convergence with regard to the ensemble size is presented. In order to assess the performance and the accuracy of the proposed method, experiments are performed making use of the atmospheric general circulation model SPEEDY. The results are compared against those obtained using the local ensemble transform Kalman filter (LETKF). Tests are performed for dense observations (100% and 50% of the model comp...

Improving the Graphical Lasso Estimation for the Precision Matrix Through Roots of the Sample Covariance Matrix

ABSTRACTIn this article, we focus on the estimation of a high-dimensional inverse covariance (i.e., precision) matrix. We propose a simple improvement of the graphical Lasso (glasso) framework that is able to attain better statistical performance without increasing significantly the computational cost. The proposed improvement is based on computing a root of the sample covariance matrix to reduce the spread of the associated eigenvalues. Through extensive numerical results, using both simulated and real datasets, we show that the proposed modification improves the glasso procedure. Our results reveal that the square-root improvement can be a reasonable choice in practice. Supplementary material for this article is available online.

Inferences in panel data with interactive effects using large covariance matrices

We consider efficient estimation of panel data models with interactive effects, which relies on a high-dimensional inverse covariance matrix estimator. By using a consistent estimator of the error covariance matrix, we can take into account both cross-sectional correlations and heteroskedasticity. In the presence of cross-sectional correlations, the proposed estimator eliminates the cross-sectional correlation bias, and is more efficient than the existing methods. The rate of convergence is also improved. In addition, we find that when the statistical inference involves estimating a high-dimensional inverse covariance matrix, the minimax convergence rate on large covariance estimations is not sufficient for inferences. To address this issue, a new “doubly weighted convergence” result is developed. The proposed method is applied to the US divorce rate data. We find that our more efficient estimator identifies the significant effects of divorce-law reforms on the divorce rate, and provides tighter confidence intervals than existing methods. This provides a confirmation for the empirical findings of Wolfers (2006) under more general unobserved heterogeneity.

INCREASE OF EFFICIENCY OF SUPPRESSION OF THE ACTIVE NOISE JAMMING IN RADAR INTERCEPTION OF AIR TARGETS IN THE REVIEW MODE

The modified algorithm of the adaptive compensating of the active noise jamming, based on usage of the recurrence estimation of inverse covariance matrix of jamming in the course of an self-tuning is considered. The matrix estimation is formed by a method of a serial regression (SRM), a consequence application of a lemma about inversing of a matrix to differential – to the difference equation for a rounded estimation of direct covariance matrix of jamming (CMJ). The single estimation of straight line CMJ is formed as result dyad products of the instant vectors of noises in handling channels. In partial channels the adaptive transversal filters with the self-tuning on value of a signal of an aggregate error are implemented.In article skeleton diagrams of devices of the compensating implemented by various methods are resulted. Mathematical modeling of operation of the algorithms, implemented by a method of the least squares (criterion of a minimum RMS error) and a method of a serial regression is performed.Results of mathematical modeling of operation of the device of compensating of active noises jamming in the conditions of the active radio-electronic counteraction at multipath propagation of signals both non-identical amplitude and phase characteristics of channels of handling are resulted. Optimal values of technical parameters of devices of compensating of the active noises jamming providing obtaining of demanded values of coefficient of suppression are given. The comparative analysis of indexes of efficiency of classical and offered algorithms of compensating is made. Advantage of algorithm of compensating of the active noises jamming, using the recurrence estimation of a inverse covariance matrix of jamming is shown.The algorithm of functioning on the basis of SRM allows to lower essentially requirements to specialized VLSI (FPGA) at implementation is hardware – program complexes for preprocessing of the radar, radio communication and radio – navigation information in the conditions of influence of the active noises jamming in the presence of effects of multipath propagation of signals, in the presence of delay of signals on an antenna system aperture, and also at unremovable technological dispersion amplitude and phase characteristics of the microwave of channels of handling.

Practical inexact proximal quasi-Newton method with global complexity analysis

Recently several methods were proposed for sparse optimization which make careful use of second-order information (Hsieh et al. in Sparse inverse covariance matrix estimation using quadratic approximation. In: NIPS, 2011; Yuan et al. in An improved GLMNET for l1-regularized logistic regression and support vector machines. National Taiwan University, Taipei City, 2011; Olsen et al. in Newton-like methods for sparse inverse covariance estimation. In: NIPS, 2012; Byrd et al. in A family of second-order methods for convex l1-regularized optimization. Technical report, 2012) to improve local convergence rates. These methods construct a composite quadratic approximation using Hessian information, optimize this approximation using a first-order method, such as coordinate descent and employ a line search to ensure sufficient descent. Here we propose a general framework, which includes slightly modified versions of existing algorithms and also a new algorithm, which uses limited memory BFGS Hessian approximations, and provide a novel global convergence rate analysis, which covers methods that solve subproblems via coordinate descent.

Robust and Sparse Estimation of the Inverse Covariance Matrix Using Rank Correlation Measures

Spearman's rank correlation is a robust alternative for the standard correlation coefficient. By using ranks instead of the actual values of the observations, the impact of outliers remains limited. In this paper, we study an estimator based on this rank correlation measure for estimating covariance matrices and their inverses. The resulting estimator is robust and consistent at the normal distribution. By applying the graphical lasso, the inverse covariance matrix estimator is positive definite if more variables than observations are available in the data set. Moreover, it will contain many zeros, and is therefore said to be sparse. Instead of Spearman's rank correlation, one can use the Quadrant correlation or Gaussian rank scores. A simulation study compares the different estimators. This type of estimator is particularly useful for estimating (inverse) covariance matrices in high dimensions, when the data may contain several outliers in many cells of the data matrix. More traditional robust estimators are not well defined or computable in this setting. An important feature of the proposed estimators is their simplicity and easiness to compute using existing software.

Toward optimal model averaging in regression models with time series errors

Consider a regression model with infinitely many parameters and time series errors. We are interested in choosing weights for averaging across generalized least squares (GLS) estimators obtained from a set of approximating models. However, GLS estimators, depending on the unknown inverse covariance matrix of the errors, are usually infeasible. We therefore construct feasible generalized least squares (FGLS) estimators using a consistent estimator of the unknown inverse matrix. Based on this inverse covariance matrix estimator and FGLS estimators, we develop a feasible autocovariance-corrected Mallows model averaging criterion to select weights, thereby providing an FGLS model averaging estimator of the true regression function. We show that the generalized squared error loss of our averaging estimator is asymptotically equivalent to the minimum one among those of GLS model averaging estimators with the weight vectors belonging to a continuous set, which includes the discrete weight set used in Hansen (2007) as its proper subset.

A joint convex penalty for inverse covariance matrix estimation

The paper proposes a joint convex penalty for estimating the Gaussian inverse covariance matrix. A proximal gradient method is developed to solve the resulting optimization problem with more than o...

A joint convex penalty for inverse covariance matrix estimation

The paper proposes a joint convex penalty for estimating the Gaussian inverse covariance matrix. A proximal gradient method is developed to solve the resulting optimization problem with more than one penalty constraints. The analysis shows that imposing a single constraint is not enough and the estimator can be improved by a trade-off between two convex penalties. The developed framework can be extended to solve wide arrays of constrained convex optimization problems. A simulation study is carried out to compare the performance of the proposed method to graphical lasso and the SPICE estimate of the inverse covariance matrix.

A small review and further studies on the LASSO

Abstract High-dimensional data analysis arises from almost all scienti c areas, evolving withdevelopment of computing skills, and has encouraged penalized estimations that playimportant roles in statistical learning. For the past years, various penalized estimationshave been developed, and the least absolute shrinkage and selection operator (LASSO)proposed by Tibshirani (1996) has shown outstanding ability, earning the rst place onthe development of penalized estimation. In this paper, we rst introduce a number ofrecent advances in high-dimensional data analysis using the LASSO. The topics includevarious statistical problems such as variable selection and grouped or structured variableselection under sparse high-dimensional linear regression models. Several unsupervisedlearning methods including inverse covariance matrix estimation are presented. In ad-dition, we address further studies on new applications which may establish a guidelineon how to use the LASSO for statistical challenges of high-dimensional data analysis.Keywords: High dimension, LASSO, penalized estimation, review.

Bootstrap—An exploration

We review a few unusual aspects of Bootstrap and some of the recent theoretical as well as methodological advances. We discuss the handling of non-linearity by Bootstrap through a numerical example in Section 2. Application to the estimation of high-dimensional inverse covariance matrix is presented in Section 3 with emphasis on the Augmented Bootstrap and a Bayesian version of it. Another high dimensional example, namely, Random Forest and its offshoot random survival forest (Ishwaran et al. (2008) [32]) are discussed in Section 4. Bootstrap for massive data, introduced by Kleiner et al. (2011) [35], is discussed in Section 4. In Section 5, we discuss some aspects of Bootstrap in the context of hypothesis testing in high-dimension.

A tutorial on the Lasso approach to sparse modeling

In applied research data are often collected from sources with a high dimensional multivariate output. Analysis of such data is composed of e.g. extraction and characterization of underlying patterns, and often with the aim of finding a small subset of significant variables or features. Variable and feature selection is well-established in the area of regression, whereas for other types of models this seems more difficult. Penalization of the L1 norm provides an interesting avenue for such a problem, as it produces a sparse solution and hence embeds variable selection. In this paper a brief introduction to the mathematical properties of using the L1 norm as a penalty is given. Examples of models extended with L1 norm penalties/constraints are presented. The examples include PCA modeling with sparse loadings which enhance interpretability of single components. Sparse inverse covariance matrix estimation is used to unravel which variables are affecting each other, and a modified PCA to model data with (piecewise) constant responses in e.g. process monitoring is shown. All examples are demonstrated on real or synthetic data. The results indicate that sparse solutions, when appropriate, can enhance model interpretability.

Simultaneous multiple response regression and inverse covariance matrix estimation via penalized Gaussian maximum likelihood

Multivariate regression is a common statistical tool for practical problems. Many multivariate regression techniques are designed for univariate response cases. For problems with multiple response variables available, one common approach is to apply the univariate response regression technique separately on each response variable. Although it is simple and popular, the univariate response approach ignores the joint information among response variables. In this paper, we propose three new methods for utilizing joint information among response variables. All methods are in a penalized likelihood framework with weighted L1 regularization. The proposed methods provide sparse estimators of the conditional inverse covariance matrix of the response vector, given explanatory variables as well as sparse estimators of regression parameters. Our first approach is to estimate the regression coefficients with plug-in estimated inverse covariance matrices, and our second approach is to estimate the inverse covariance matrix with plug-in estimated regression parameters. Our third approach is to estimate both simultaneously. Asymptotic properties of these methods are explored. Our numerical examples demonstrate that the proposed methods perform competitively in terms of prediction, variable selection, as well as inverse covariance matrix estimation.

Simultaneous multiple response regression and inverse covariance matrix estimation via penalized Gaussian maximum likelihood

Simultaneous multiple response regression and inverse covariance matrix estimation via penalized Gaussian maximum likelihood