Covariance Selection Problem Research Articles

Topological regularization with information filtering networks

This paper introduces a novel methodology to perform topological regularization in multivariate probabilistic modeling by using sparse, complex, networks which represent the system’s dependency structure and are called information filtering networks (IFN). This methodology can be directly applied to covariance selection problem providing an instrument for sparse probabilistic modeling with both linear and non-linear multivariate probability distributions such as the elliptical and generalized hyperbolic families. It can also be directly implemented for topological regularization of multicollinear regression. In this paper, I describe in detail an application to sparse modeling with multivariate Student-t. A specific expectation–maximization likelihood maximization procedure over a sparse chordal network representation is proposed for this sparse Student-t case. Examples with real data from stock prices log-returns and from artificially generated data demonstrate applicability, performances, robustness and potentials of this methodology.

Fast algorithms for sparse inverse covariance estimation

ABSTRACTSparse precision matrix (i.e. inverse covariance matrix in statistic term) estimation is an important problem in many applications of multivariate analysis. The problem becomes very challenging when the dimension of data is much larger than the number of samples. In this paper, we propose a convex relaxation model for the sparse covariance selection problem, which is solved by the well-known alternating direction method of multipliers (ADMM). Furthermore, a new model with positive semi-definite constraint is proposed. Numerical results show that the ADMM-based methods perform favourably compared with the column-wise manner on both synthetic and real data.

A Proximal Point Algorithm for Log-Determinant Optimization with Group Lasso Regularization

We consider the covariance selection problem where variables are clustered into groups and the inverse covariance matrix is expected to have a blockwise sparse structure. This problem is realized via penalizing the maximum likelihood estimation of the inverse covariance ma- trix by group Lasso regularization. We propose to solve the resulting log-determinant optimization problem with the classical proximal point algorithm (PPA). At each iteration, as it is difficult to update the primal variables directly, we first solve the dual subproblem by an inexact semismooth Newton-CG method and then update the primal variables by explicit formulas based on the com- puted dual variables. We also propose to accelerate the PPA by an inexact generalized Newton's method when the iterate is close to the solution. Theoretically, we prove that at the optimal solution, the nonsingularity of the generalized Hessian matrices of the dual subproblem is equivalent to the constraint nondegeneracy condition for the primal problem. Global and local convergence results are also presented for the proposed PPA. Moreover, based on the augmented Lagrangian function of the dual problem we derive an alternating direction method (ADM), which is easily implementable and is demonstrated to be efficient for random problems. Numerical results, including comparisons with the ADM on both synthetic and real data, are presented to demonstrate that the proposed Newton-CG based PPA is stable and efficient and, in particular, outperforms the ADM when high accuracy is required.

Ultranet: efficient solver for the sparse inverse covariance selection problem in gene network modeling

Graphical Gaussian models (GGMs) are a promising approach to identify gene regulatory networks. Such models can be robustly inferred by solving the sparse inverse covariance selection (SICS) problem. With the high dimensionality of genomics data, fast methods capable of solving large instances of SICS are needed. We developed a novel network modeling tool, Ultranet, that solves the SICS problem with significantly improved efficiency. Ultranet combines a range of mathematical and programmatical techniques, exploits the structure of the SICS problem and enables computation of genome-scale GGMs without compromising analytic accuracy. Ultranet is implemented in C++ and available at www.broadinstitute.org/ultranet.

Covariance selection and multivariate dependence

Considering the covariance selection problem of multivariate normal distributions, we show that its Fenchel dual formulation is insightful and allows one to calculate direct estimates under decomposable models. We next generalize the covariance selection to multivariate dependence, which includes MTP2 and trends in longitudinal studies as special cases. The iterative proportional scaling algorithm, used for estimation in covariance selection problems, may not lead to the correct solution under such dependence. Addressing this situation, we present a new algorithm for dependence models and show that it converges correctly using tools from Fenchel duality. We discuss the speed of convergence of the new algorithm. When normality does not hold, we show how to estimate the covariance matrix in an empirical entropy approach. The approaches are compared via simulation and it is shown that the estimators developed here compare favorably with existing ones. The methodology is applied on a real data set involving decreasing CD4+ cell numbers from an AIDS study.

Alternating Direction Method for Covariance Selection Models

The covariance selection problem captures many applications in various fields, and it has been well studied in the literature. Recently, an l 1-norm penalized log-likelihood model has been developed for the covariance selection problem, and this novel model is capable of completing the model selection and parameter estimation simultaneously. With the rapidly increasing magnitude of data, it is urged to consider efficient numerical algorithms for large-scale cases of the l 1-norm penalized log-likelihood model. For this purpose, this paper develops the alternating direction method (ADM). Some preliminary numerical results show that the ADM approach is very efficient for large-scale cases of the l 1-norm penalized log-likelihood model.

A block coordinate gradient descent method for regularized convex separable optimization and covariance selection

We consider a class of unconstrained nonsmooth convex optimization problems, in which the objective function is the sum of a convex smooth function on an open subset of matrices and a separable convex function on a set of matrices. This problem includes the covariance selection problem that can be expressed as an l 1-penalized maximum likelihood estimation problem. In this paper, we propose a block coordinate gradient descent method (abbreviated as BCGD) for solving this class of nonsmooth separable problems with the coordinate block chosen by a Gauss-Seidel rule. The method is simple, highly parallelizable, and suited for large-scale problems. We establish global convergence and, under a local Lipschizian error bound assumption, linear rate of convergence for this method. For the covariance selection problem, the method can terminate in $${O(n^3/\epsilon)}$$ iterations with an $${\epsilon}$$ -optimal solution. We compare the performance of the BCGD method with the first-order methods proposed by Lu (SIAM J Optim 19:1807–1827, 2009; SIAM J Matrix Anal Appl 31:2000–2016, 2010) for solving the covariance selection problem on randomly generated instances. Our numerical experience suggests that the BCGD method can be efficient for large-scale covariance selection problems with constraints.