Dual spectral projected gradient method for generalized log-det semidefinite programming

Graphical models have become a common tool in many fields and a useful way of modeling probability distributions. In this thesis, we investigate approaches for graph estimation and density estimation problems in high dimensions. The estimation and model selection problems in Gaussian graphical models are equivalent to the estimation of precision matrix and identification of its zero-pattern. In Chapter 2, we propose a new estimation method by considering a convex combination of a set of individual estimators with various sparsity patterns. We analyze the risk of this aggregation estimator and show by an oracle that it is comparable to the risk of the best estimator based on a single graph. In Chapter 3, we investigate robust methods for Gaussian graphical models in the presence of possible outliers and corrupted data. We consider the neighborhood selection and graphical lasso algorithms and show that the robust counterparts obtained using the trimmed inner product or the nonparanormal give stronger performance guarantees. Gaussian graphical models maintain tractable inference, but they are limited in their ability to flexibly model the bivariate and higher order marginals. In Chapter 4, we study tree-based graphical models and develop new density estimation approaches under structural constraints---scale-free network and shared edges among multiple graphs. Our methods arise from a Bayesian formulation as the MAP estimates and solve the optimization problems via a minorize-maximization procedure with Kruskal's algorithm. The ability of tree-based graphical models to model complex independence graphs is compromised. In Chapter 5, we combine the ideas behind Gaussian graphical models and tree-based graphical models to form a new nonparametric family of graphical models, which relax the normality assumption and increase statistical efficiency by modeling the forest with kernel density estimators and modeling each blossom with the nonparanormal. Our analysis and experimental results indicate that the newly proposed methods in this thesis can be powerful alternatives to standard approaches for graph estimation and density estimation problems.

On skewed Gaussian graphical models

Loop-based conic multivariate adaptive regression splines is a novel method for advanced construction of complex biological networks

In this paper, we consider a least square semidefinite programming problem under ellipsoidal data uncertainty. We show that the robustification of this uncertain problem can be reformulated as a semidefinite linear programming problem with an additional second-order cone constraint. We then provide an explicit quantitative sensitivity analysis on how the solution under the robustification depends on the size/shape of the ellipsoidal data uncertainty set. Next, we prove that, under suitable constraint qualifications, the reformulation has zero duality gap with its dual problem, even when the primal problem itself is infeasible. The dual problem is equivalent to minimizing a smooth objective function over the Cartesian product of second-order cones and the Euclidean space, which is easy to project onto. Thus, we propose a simple variant of the spectral projected gradient method (Birgin et al. in SIAM J. Optim. 10:1196---1211, 2000) to solve the dual problem. While it is well-known that any accumulation point of the sequence generated from the algorithm is a dual optimal solution, we show in addition that the dual objective value along the sequence generated converges to a finite value if and only if the primal problem is feasible, again under suitable constraint qualifications. This latter fact leads to a simple certificate for primal infeasibility in situations when the primal feasible set lies in a known compact set. As an application, we consider robust correlation stress testing where data uncertainty arises due to untimely recording of portfolio holdings. In our computational experiments on this particular application, our algorithm performs reasonably well on medium-sized problems for real data when finding the optimal solution (if exists) or identifying primal infeasibility, and usually outperforms the standard interior-point solver SDPT3 in terms of CPU time.

Gaussian Graphical Models Identify Networks of Dietary Intake in a German Adult Population

Though Gaussian graphical models have been widely used in many scientific fields, relatively limited progress has been made to link graph structures to external covariates. We propose a Gaussian graphical regression model, which regresses both the mean and the precision matrix of a Gaussian graphical model on covariates. In the context of co-expression quantitative trait locus (QTL) studies, our method can determine how genetic variants and clinical conditions modulate the subject-level network structures, and recover both the population-level and subject-level gene networks. Our framework encourages sparsity of covariate effects on both the mean and the precision matrix. In particular for the precision matrix, we stipulate simultaneous sparsity, that is, group sparsity and element-wise sparsity, on effective covariates and their effects on network edges, respectively. We establish variable selection consistency first under the case with known mean parameters and then a more challenging case with unknown means depending on external covariates, and establish in both cases the convergence rates and the selection consistency of the estimated precision parameters. The utility and efficacy of our proposed method is demonstrated through simulation studies and an application to a co-expression QTL study with brain cancer patients. Supplementary materials for this article are available online.

The Gaussian graphical model (GGM) is a probabilistic modelling approach used in the system biology to represent the relationship between genes with an undirected graph. In graphical models, the genes and their interactions are denoted by nodes and the edges between nodes. Hereby, in this model, it is assumed that the structure of the system can be described by the inverse of the covariance matrix, \(\varTheta \), which is also called as the precision, when the observations are formulated via a lasso regression under the multivariate normality assumption of states. There are several approaches to estimate \(\varTheta \) in GGM. The most well-known ones are the neighborhood selection algorithm and the graphical lasso (glasso) approach. On the other hand, the multivariate adaptive regression splines (MARS) is a non-parametric regression technique to model nonlinear and highly dependent data successfully. From previous simulation studies, it has been found that MARS can be a strong alternative of GGM if the model is constructed similar to a lasso model and the interaction terms in the optimal model are ignored to get comparable results with respect to the GGM findings. Moreover, it has been detected that the major challenge in both modelling approaches is the high sparsity of \(\varTheta \) due to the possible non-linear interactions between genes, in particular, when the dimensions of the networks are realistically large. In this study, as the novelty, we suggest the Bernstein operators, namely, Bernstein and Szasz polynomials, in the raw data before any lasso type of modelling and associated inference approaches. Because from the findings via GGM with small and moderately large systems, we have observed that the Bernstein polynomials can increase the accuracy of the estimates. Hence, in this work, we perform these operators firstly into the most well-known inference approaches used in GGM under realistically large networks. Then, we investigate the assessment of these transformations for the MARS modelling as the alternative of GGM again under the same large complexity. By this way, we aim to propose these transformation techniques for all sorts of modellings under the steady-state condition of the protein-protein interaction networks in order to get more accurate estimates without any computational cost. In the evaluation of the results, we compare the precision and F-measures of the simulated datasets.

A general algorithm for covariance modeling of discrete data

This paper introduces a two-step method to exploit hidden block structures in Gaussian graphical models, achieving improved accuracy and scalability with theoretical guarantees and strong empirical performance. This paper addresses the estimation of Gaussian graphical models with block-diagonal precision matrices. Due to the substantial presence of zero off-diagonal entries, accurately identifying block structures becomes challenging. To tackle this, we introduce a two-step method called Cluster Gelnet (CG). First, we employ the K-medoids clustering algorithm to detect block structures within the precision matrix. Subsequently, we estimate the graphical model individually within each detected block. By partitioning the estimation into smaller blocks, CG significantly enhances computational efficiency and accuracy. Theoretical guarantees for the proposed method are provided, and simulation studies demonstrate its superior performance compared to existing methods. We also apply CG to analyze a breast cancer microarray dataset, identifying three important genes and discussing their potential roles.

In this paper, we introduce a transformation that converts a class of linear and nonlinear semidefinite programming (SDP) problems into nonlinear optimization problems. For those problems of interest, the transformation replaces matrix-valued constraints by vector-valued ones, hence reducing the number of constraints by an order of magnitude. The class of transformable problems includes instances of SDP relaxations of combinatorial optimization problems with binary variables as well as other important SDP problems. We also derive gradient formulas for the objective function of the resulting nonlinear optimization problem and show that both function and gradient evaluations have affordable complexities that effectively exploit the sparsity of the problem data. This transformation, together with the efficient gradient formulas, enables the solution of very large-scale SDP problems by gradient-based nonlinear optimization techniques. In particular, we propose a first-order log-barrier method designed for solving a class of large-scale linear SDP problems. This algorithm operates entirely within the space of the transformed problem while still maintaining close ties with both the primal and the dual of the original SDP problem. Global convergence of the algorithm is established under mild and reasonable assumptions.

Graphical models provide an effective way to reveal complicated associations in data and especially to learn the structures among large numbers of variables with respect to few observations in a high-dimensional space. In this paper, a novel graphical algorithm that integrates the dynamic time warping (DTW)-D measure into the birth–death Markov Chain Monte Carlo (BDMCMC) methodology (DTWD-BDMCMC) is proposed for modeling the intrinsic correlations buried in data. The DTW-D, which is the ratio of DTW over the Euclidean distance (ED), is targeted to calibrate the warping observation sequences. The approach of the BDMCMC is a Bayesian framework used for structure learning in sparse graphical models. In detail, a modified DTW-D distance matrix is first developed to construct a weighted covariance instead of the traditional covariance calculated with the ED. We then build on Bayesian Gaussian models with the weighted covariance with the aim to be robust against problems of sequence distortion. Moreover, the weighted covariance is used as limited prior information to facilitate an initial graphical structure, on which we finally employ the BDMCMC for the determination of the reconstructed Gaussian graphical model. This initialization is beneficial to improve the convergence of the BDMCMC sampling. We implement our method on broad simulated data to test its ability to deal with different kinds of graphical structures. This paper demonstrates the effectiveness of the proposed method in comparison with its rivals, as it is competitively applied to Gaussian graphical models and copula Gaussian graphical models. In addition, we apply our method to explore real-network attacks and genetic expression data.

The semi definite programming (SDP) problem is one of the central problems in mathematical optimization. The primal-dual interior-point method (PDIPM) is one of the most powerful algorithms for solving SDP problems, and many research groups have employed it for developing software packages. However, two well-known major bottlenecks, i.e., the generation of the Schur complement matrix (SCM) and its Cholesky factorization, exist in the algorithmic framework of the PDIPM. We have developed a new version of the semi definite programming algorithm parallel version (SDPARA), which is a parallel implementation on multiple CPUs and GPUs for solving extremely large-scale SDP problems with over a million constraints. SDPARA can automatically extract the unique characteristics from an SDP problem and identify the bottleneck. When the generation of the SCM becomes a bottleneck, SDPARA can attain high scalability using a large quantity of CPU cores and some processor affinity and memory interleaving techniques. SDPARA can also perform parallel Cholesky factorization using thousands of GPUs and techniques for overlapping computation and communication if an SDP problem has over two million constraints and Cholesky factorization constitutes a bottleneck. We demonstrate that SDPARA is a high-performance general solver for SDPs in various application fields through numerical experiments conducted on the TSUBAME 2.5 supercomputer, and we solved the largest SDP problem (which has over 2.33 million constraints), thereby creating a new world record. Our implementation also achieved 1.713 PFlops in double precision for large-scale Cholesky factorization using 2,720 CPUs and 4,080 GPUs.

In this talk, we present our ongoing research project. The objective of this project is to develop advanced computing and optimization infrastructures for extremely large-scale graphs on post peta-scale supercomputers. We explain our challenge to Graph 500 and Green Graph 500 benchmarks that are designed to measure the performance of a computer system for applications that require irregular memory and network access patterns. The 1st Graph500 list was released in November 2010. The Graph500 benchmark measures the performance of any supercomputer performing a BFS (Breadth-First Search) in terms of traversed edges per second (TEPS). In 2014 and 2015, our project team was a winner of the 8th, 10th, and 11th Graph500 and the 3rd to 6th Green Graph500 benchmarks, respectively. We also present our parallel implementation for large-scale SDP (SemiDefinite Programming) problem. The semidefinite programming (SDP) problem is a predominant problem in mathematical optimization. The primal-dual interior-point method (PDIPM) is one of the most powerful algorithms for solving SDP problems, and many research groups have employed it for developing software packages. We solved the largest SDP problem (which has over 2.33 million constraints), thereby creating a new world record. Our implementation also achieved 1.774 PFlops in double precision for large-scale Cholesky factorization using 2,720 CPUs and 4,080 GPUs on the TSUBAME 2.5 supercomputer.

Loss function, unbiasedness, and optimality of Gaussian graphical model selection

BackgroundIn systems biology, it is important to reconstruct regulatory networks from quantitative molecular profiles. Gaussian graphical models (GGMs) are one of the most popular methods to this end. A GGM consists of nodes (representing the transcripts, metabolites or proteins) inter-connected by edges (reflecting their partial correlations). Learning the edges from quantitative molecular profiles is statistically challenging, as there are usually fewer samples than nodes (‘high dimensional problem’). Shrinkage methods address this issue by learning a regularized GGM. However, it remains open to study how the shrinkage affects the final result and its interpretation.ResultsWe show that the shrinkage biases the partial correlation in a non-linear way. This bias does not only change the magnitudes of the partial correlations but also affects their order. Furthermore, it makes networks obtained from different experiments incomparable and hinders their biological interpretation. We propose a method, referred to as ‘un-shrinking’ the partial correlation, which corrects for this non-linear bias. Unlike traditional methods, which use a fixed shrinkage value, the new approach provides partial correlations that are closer to the actual (population) values and that are easier to interpret. This is demonstrated on two gene expression datasets from Escherichia coli and Mus musculus.ConclusionsGGMs are popular undirected graphical models based on partial correlations. The application of GGMs to reconstruct regulatory networks is commonly performed using shrinkage to overcome the ‘high-dimensional problem’. Besides it advantages, we have identified that the shrinkage introduces a non-linear bias in the partial correlations. Ignoring this type of effects caused by the shrinkage can obscure the interpretation of the network, and impede the validation of earlier reported results.